elmanto.github.io

View on GitHub

Getting a Ph.D. in System Security – the FAQ

I remember when I was a master’s student, unsure about the path to take, and surrounded by questions like “Is the Ph.D. a good road?” or some months later, after my first papers rejected, “Did I choose right?”. Once, an old professor told me something like, “doing a Ph.D. is like a 3-year vacation”. Probably to some extent, or in some cases, this sentence could be plausible, but more likely, I guess this was an appealing way to convince young students to start a career in academia.

This blog post arises from here, from the lack of information about this road and wants to describe my personal experience demystifying some common legends and answering some questions that future/current Ph.D. students may have. I don’t pretend that my point of view is shared among all of you and similarly that many people read this, but my hope is to give a hint to all students uncertain about what to do with their life after their Master’s Degree or during the academic period. Thus I will go through some points that I consider quite useful and informative, and for each of them, I’ll highlight what my approach looks like. I point out that the choices I made don’t necessarily have to be considered correct, and it is very likely that the correctness depends on several other factors such as your interests, final goal, lifestyle, etc. But I opted to tell the story ``as-is’’, describing my personal point of view and collecting information just from one source: my direct experience. Finally, note that many of the aspects I will describe are related to the topic of my research, which is system security. While some points could be common with other research areas, honestly, I do not know how much overlapping there can be between security and other fields, which are very far.

Before starting

My starting point

When I started, one of the first things I noticed was the different backgrounds between me and the other doctoral students. Indeed I individuated three main scenarios. Some students start the 3-year academic period with a super-strong background on their topic. Others, instead, know very basic concepts related to the field of interest but have good skills that allow them to overcome the initial difficulties (e.g., super good at coding). Somewhere between these two configurations, there was me.

The university where I came from (University of Genoa) is definitely not a top one in Italy, and at the same time, it is not famous for providing a super-strong background in system security. During the MSc, I took only ONE security class and ONE (very generic) operating system class while I believe these are the two fundamental subjects in my research area. Luckily I had the opportunity to meet some skilled people (mostly the Ph.D.s of my old university) during my bachelor’s and master’s degrees that helped me to move my first steps in security, for instance, by introducing me to the CTF world or pointing me to some fundamental concepts of the field. With a bit of personal effort and some hints from the right people, in the last year of master’s degree, I had some basic knowledge about several topics such as vulnerability detection and exploitation in simple web/C applications as well as reversing of binary code. Nothing special, but this allowed me to apply for a Ph.D. position and be considered eligible for that.

Is there anything I can do to understand if I like the Ph.D. life?

Try it yourself. Probably the best thing that you can do is to face a research problem alone, no matter if it gets published or not. For instance, in my case, I focused my master thesis on a research project that luckily, in the end, became a top-tier publication, even though obviously at that time I was not sure about the research direction. Understanding how the public research world works helped me a lot to realize that I really loved that activity. This was probably one of the main motivations behind the choice of starting the Ph.D.

During the Ph.D.

In this central chapter I emphasize several aspects that are related to the actual 3-year Ph.D. period, ranging from background factors such as the environment, human aspects, etc. to more fundamental aspects like the approach I had during this period or the topic choice.

The topic choice

One of the first choices that novice doctoral students have to make is related to the main topic they will be focusing on for the upcoming three years. Although this is somehow a trivial consideration, let me say that this choice is extremely important. The good news is that overall, picking one topic does not exclude the chance to do research in other fields in your area. For instance, while the main topic of my research was theoretically focusing on human-in-the-loop approaches for binary analysis, in parallel, I could experiment with different research threads such as malware analysis and fuzzing. The bad news is that in the end, you will be required to write a thesis on your topic, which is way easier if all your papers are linked to the same root subject. Also, while IMHO it completely makes sense to be ``curious’’ at this time, and explore different research areas that may be not strongly related with your thesis topic, I suggest you to choose a subject that motivates you and in which you feel real interest. This will help you to keep working on it for the full duration of your student status and to cultivate your interest in it.

Honestly, I am curious by nature and I never felt that my thesis topic was not sufficiently motivating me to push hard with my work. But there are papers and papers and each one deserves a specific judgment. For instance, among the three papers in the final thesis document, one was a human study (i.e., a study that involves experiments with some users to investigate how humans perform a certain activity) that I loved and I am very proud of. However, inherently I like technical challenges, and thus I opted to skip other potential human study papers that could fit in the thesis because I preferred to seek different ones combining technical complexity as well as a close relationship to the topic. This had the effect of reducing the search space for my candidate papers and, in the end, resulted in some complications of the thesis plot, but on the other hand, this allowed me to understand better what I like more and what instead is a bit too far to represent the core of my job. If I came back, I think I would do the same in the end, with no regrets because I am very proud of all papers that I contributed. Just, I noticed there are some topics that are more stimulating than others. This is also a cool thing about the Ph.D., you develop some new interests in some previously-unknown topics, and you discover some subjects don’t worth investigating for you.

The horizontal vs vertical dilemma

Related to the previous chapter on the subject choice, I want to speak about what I named the “horizontal vs. vertical” dilemma, one of the mental trips that affected my mind during the last three years. Put it simply, one can decide to move only “vertically”, exploring very deep and nested aspects of his/her topic and becoming a very specialized expert in that context. The alternative way instead, is to move “horizontally” and work on subjects that are not necessarily a specification of the parent one. As you may have guessed, I belong to the second category, even though for all three years I had doubt about what was the best choice. For instance, I wondered very often which strategy was the best to look for a job in a second moment. However, this is the classical moment where you make a decision and follow it. Then what was my decision based on? Well, what made the difference for me was the freedom in my research activity to consider different areas. This is why I don’t have regrets, and I would re-do the same.

This ofc does not imply that the vertical approach is wrong, and honestly I really appreciate people who followed this direction and focused on a single topic for the entire duration of the Ph.D. Nevertheless, I feel satisfied with my choice, and I believe this is too personal to give suggestions. Just, be aware that both solutions are valid and bring pros/cons, and your final choice should be what makes you feel better. My personal suggestion is that unless you start a Ph.D. with a very specific goal (“I want to work on exactly this specific X aspect of this Y field with this Z initial assumptions”), moving “horizontally” will help you to manage and learn multiple aspects at 360 degrees. Then you’ll do your considerations and go for the top 2 or 3 arguments that you learned to love. In other cases, feel free to select the strategy you like, very probably you’ll never lose as you always have a backup plan (your starting topic).

The ``push push push’’ philosophy

Put it simply, don’t start a Ph.D. if you don’t want to work that much. If your expectation is to have an office-like schedule where every day you enter/exit at the same time this is not going to fly. IMHO, the time you dedicate on a project should be ``goal-dependent’’. At least in an initial phase, it totally makes sense that you go relatively slow. If you want to gift yourself with some free days, this is the right moment. But whenever you start to collect interesting results, well, that is the moment to change mindset and start to push and accelerate to conclude that project ASAP. Still, this doesn’t mean you cannot enjoy your free time. But let’s say that there will be moments to relax and moments to work. And in these last moments, you should be very focused on your goals.

You may wonder why. My answer is that a Ph.D. is, at least in part, a period you dedicate to study and grow as a technical individual. Also, this is probably the last period you have in your life to study what you want. Whenever you finish that, you will have two possibilities: industry and academia. In the former case, probably same-age people already started to work 3 years before. You have to be very well-prepared if you want to work in research or other industry positions. In the latter case, you have to push to simply maximize your papers, that is what will definitely help you to become, one day, a professor.

TBH, one of the few things I would change about my Ph.D. is that at least during the first year I didn’t push that much. When you start you think that you will have enough time for sure and you can afford some reduced-speed days. ERROR! It turns out that time is precious especially for a Ph.D. where the final thesis defense can happen only if you have a sufficient number of papers. Moreover, wasted time is also a damage to your intrinsic learning activity that is what in the end will pay out your time spent in the laboratory of a certain university. Finally, as a more practical point of view, I believe it is much better to enjoy your free time after your work is done rather than the opposite.

The learning process

Another reason that motivated me was to spend a proper amount of time to learn new things because I felt that my level of preparation was unsufficient after the Master degree, at least for my personal standards. While studying new concepts is not a thing you can do exclusively during a Ph.D., I believe this period is particularly suitable to learn novel approaches/technologies/tools/theories/etc. Now, what happened in my experience is that I learned things from two separate sources. Of course, doing research and experimenting with custom approaches to solve a certain security problem was the first one and it helped me a lot to learn new things on-the-fly. You can play with very different technologies and techniques that will become part of your background and at the same time you will be ready to increase your knowledge level about a specific topic.

On the other hand, I would not be the same person if I hadn’t studied ``stuff’’ on my own. Obviously, not random stuff, but just what you exactly like. For instance, what helped me was playing CTFs, an educational approach that taught me many things and led me to understand what I liked more, other directions to improve, ..

On top of this, I had the possibility to dedicate my time exactly on what I loved. One month I wanted to study nested operating system concepts and I had time for this. Another month I was studying compilers. Yet another one I was focusing on a new programming language. And this was not strongly required by my papers, was just for personal satisfaction and curiosity.

This has also the advantage that you will conclude the Ph.D. with a new skill, that is, “learn to learn”. When you get used to study things, grep in the source code, look for materials, well you’ll become faster to approach also very different and complicated problems. But before “learning to learn” try to be sure, at the beginning of the Ph.D. that one of your interests is actually “learning”. If you’re not curious, then maybe the Ph.D. is not for you.

The academic workflow

For those who are not familiar with the academic workflow, I’ll give a short introduction here. This is needed to understand the following sections, especially when I deal with topics such as “reviewers” or “accepted paper”. Let’s say you have discovered a super novel approach or performed a very interesting measurement about a specific phenomenon, eventually you will decide to write a scientific paper about it. Typically you’ll be using a specific language a.k.a. Latex (theoretically you can use also Word, but I strongly discourage this choice for questions of dignity), a system for documents writing and preparation that allows you to specify fine-grained settings to generate high-quality files, that commonly in our case are PDF files. Each paper is different and depending on several factors it can take more or less to write it down. To give you an estimation, I would say one month is needed to write a high-quality paper for a top conference (more about ``top conference’’ later) avoiding a rush and without becoming mad for the deadline. But of course exception to this rules exist, you will experience them if you start in academia.

That being said, when you start to write the paper, you will target either a conference or a journal. Each conference/journal has a ranking that determine its importance and consequently the difficulty to get the paper accepted. In security, some popular conferences are IEEE S&P, ACM CCS, Usenix and NDSS while a popular journal is TOPS. Independently on your target, a certain number of reviewers will be in charge of reading your paper, criticising it and deciding for an acceptance or a rejection of the paper. Reviewers can have different expertise levels about that specific topic, ranging from the very expert, to knowledgeable, until the security guy that knows about it but works on other security aspects. Obviously in case of acceptance you’re done with your job, whereas, in case of rejected paper, you’ll need to fix the paper to address the reviewers’ comments and submit it to a different conference. Now, the actual process is a bit more complicated, for instance there are some intermediate steps like “early-reject”, “rebuttal” and “major revision” that eventually can turn into an accepted or rejected paper. Moreover, some differencies exist between each conference/journal but at least now you have a basic understanding. Let me just add that the reviewing process is done, in most of the cases, according to a policy that we name ``double-blind’’, i.e., the paper’s authors don’t know who the reviewers are and viceversa. This, of course, to ensure that no bias exists in the reviewing mechanism.

The stress of publishing

I start by saying that each research group has different constraints in terms papers needed to defend your Ph.D.. In my research group (the S3 lab at Eurecom), the internal rule was to require 3 papers to be accepted at the time of the defense, even though very often the acceptance of the third paper could be relaxed, for instance by allowing that the third paper is under submission at the time of the defense. I totally agree that you need a way to measure the productivity of a Ph.D. student, that is necessary for several things. BUT, especially during the last year, I felt like this was not a good criteria. Indeed, unfortunately, the acceptance of a paper does not depend totally on you but instead very often the submission process introduces a combination of stress and waste of time, that lead to a delay in the fulfilment of the requirements. Probably the worst enemy of the Ph.D. student is that famous Reviewer B, that systematically rejects your paper with statements like “this work is under the bar of our top-tier conference” or “this paper lacks of this specific experiment” while you have an entire section about that. Now, since I am leaving from academia, I am not in the position to suggest a way to improve the peer-review, but what I want to underline are two things. First, to the reviewer: try to think that in several cases the paper you are rejecting is the result of a year of work of a young student that may want to receive some precious feedback. If you, dear reviewer B cannot really find any qualities (that is absolutely possible), at least be polite and behave like a real expert would do, i.e., by giving useful suggestions to young practitioners of that topic. Second, to the professors: I perfectly see you need to set a threshold at some point, to establish if a person is, or not, ready to obtain a Ph.D. as well as to justify the fact you hired a person to accomplish a certain job. But please, consider the entire path of a student, where he/she started, the topic and what he/she reached. I heard about stories of other labs in different universities where the professor presses a lot the students to maximize the number of papers. If these voices are true, as I believe, this is not fair.

W.r.t. this, I don’t have specific recommendations for future Ph.D. students, but I wanted to describe a classical problem that you may meet if you work on system security. Especially if your topic is not the hype of the moment, this may become even harder, as it will be complicated to find reviewers that know that subject. Obviously, I don’t want to say you should decide the topic depending on the difficulty of getting it accepted. But don’t forget that some topics are harder to get accepted than others. And, finally, ask about the minimum number of papers to obtain a Ph.D. in that group before starting. If it is more than one per year, and you don’t see any possibility to relax the constraint, maybe that advisor is more a manager rather than a professor. Then, as always, it’s up to you, but cope with this.

In general I am quite optimistic about my life and I was so even when my paper was rejected 4 times. A possible strategy that I can suggest for those who are not really interested at pursuing an academic career is to downgrade the conference ranking when you’re turning closer to the madness because of an high number of rejections. When I did it I got the paper instantly accepted into a slightly minor conference (AsiaCCS). Then of course, if you have time and you want to have that paper at top because i) it deserves a top conf in your opinion ii) you want it to have more visibility iii) personal satisfaction to go to top conference, then keep on submitting to the big 4, eventually you’ll get accepted.

Resiliency to bad research failure and delayed gratification

As another consequence of the academic research pipeline, I’m going to illustrate now the concept of “resiliency”. Yes, because when we experiment with a novel approach we are forced to try and re-try until we get the results we want (if we can get them). This can require from one attempt to several distinct attempts and re-implementations that internally help to form our character. This is what I mean with resiliency, and believe that it is a fundamental skill you will get in a Ph.D. and will never abandon for your following jobs, in academia and industry alike. On the other hand, this will come at the expense of a “delayed gratification”. You will be rewarded, not when you get the first good results, but when you get a paper accepted that could take much more time because as I mentioned in the paragraph before, does not depend on you. Honestly, I believe this is a reasonable price to learn a skill that would be extremely difficult to develop otherwise and that can potentially help a lot. But anyway, if you are going to start a Ph.D., be prepared to fail before succeed, make errors before correct implementations and get rejected before accepted. I know this is not fun, but it is part of the game. This is kind-of unavoidable I think. It’s unavoidable because it’s basically impossible to implement the correct approach at the first attempt and consequently getting your cool results published immediately. Therefore I cannot really suggest how my approach looked like for this sub-problem. Indeed, with this paragraph, I just want to inform you that this phenomenon exists and yes you aren’t the only one affected.

Being advised

DISCLAIMER: I had a very good relationship with my advisor and I cannot tell anything except that I learned a lot with him.

But in this section, I want to underline some patterns that I noticed in my personal experience (as well as in other student-advisor) relationships and that I find quite reasonable in the end.

Team working

Another soft skill you’ll be developing during your Ph.D. is, in several cases, the ``team working’’. Indeed, in several cases a research project has several co-authors where at least two or three co-operate to produce the results in a faster way. I believe this is very good, because you can try different configurations of your team depending on the paper. And moreover, typically you will meet people with a different background, modus operandi and other aspects such as the seniority. While you should see this as an opportunity, giving to you the chances to improve yourself and compare your way of working with other people that potentially have worked in that field for several years, there could be cases where the co-operation is made more difficult because of certain aspects. The best thing you can do is to agree with the co-authors about the code and the strategies you are following before the final implementations. It’s annoying to change your 1-month codebase after a discussion with a co-author but it’s way better to sync on this at the very beginning. Then, IMHO it’s better to split the tasks in threads, a thread for co-author. Then, everyone works on a specific part of the project and there are no annoying overlappings that could result in hazards. Finally, if the problem is more on the human side, then this is more a general question, but I believe that you should try to ignore the personal problems and involve your advisor only if the situation becomes hot.

Actual Ph.D. duration

In France this is 3 years. It can reach almost 4 years if you require an extension, for instance to conclude your thesis or to submit your last paper, but it depends. I think this depends also on where your future will be. Are you planning a long-term stay in academia? You can definitely afford to stay one year more as a Ph.D. in your lab, this will help you to learn even more subtle things about the academic life. Are you leaving from academia? Then once you’re done with the papers there is no value in extending your Ph.D., at least IMHO.

Background factors (part I): human aspects and geographical location

This is probably the most under-evaluated type of aspect: background factors. For instance speak about the environment, which includes the colleagues, the office, etc. These are extremely important to increase the productivity of an environment. There is not just work, but it’s also cool to chat about several topics with some colleagues in the morning in front of a cup of coffee. W.r.t. this, I was super-super-super lucky and I met an actual ``second’’ family at Eurecom. And this of course contributed to the feeling of being happy to reach your office in the morning to start working and abandoning it in the night after working hard for a good beer with your team. And also think about secondary factors like the office itself (are you used to working in a silent space? You can accept noise? How many virtual calls do you make per day?) as well as the geographical location. This last point also plays a certain role. Indeed, while it is important to be productive during the hours you are in front of your laptop, it is also essential to enjoy your 3-year experience at 360 degrees, including your social life outside your campus/university. Therefore, try to seek for a laboratory in a city that has some attractions. Small villages in the middle of nothing will only make you even more stressed and depressed after the first papers’ rejections.

Background factors (part II): research incentives

This is quite obvious, but to be 100% clear: you don’t start a Ph.D. for money, start a Ph.D. due to your passion and interests. Thus, look at the salary when you start to understand if this allows you to live with dignity in that region but don’t be worried if the apartment you are renting is a single-room studio as it was in my case. This is normal, you’ll have your chance to gain much more money after the Ph.D. and consider this as an investment in yourself.

In my case the net salary was approximately 1750 euros + ~150 euros of tickets to buy food. Life in the French Riviera (where EURECOM is located) is not cheap, a studio can be between 600 to 750 euros per month, and in addition you have to eat, live, and maintain a car (not mandatory, but I had it). By doing some maths, you can easily see that I didn’t become rich during the Ph.D. but in the end I could survive and still save some money.

And moreover, there are other incentives for you. Want to know what? Each conference paper accepted is one trip in a cool city (at least usually) and the university/lab typically pays entirely it. Obviously I was not that lucky because at the second year covid-19 started and lasted until the end of the third year, but you’ll be definitely more lucky than me. For instance, the only conference that I attended was NDSS, that historically takes place in San Diego. I let you imagine how carefully I attended the conference ;)

After the Ph.D.

Maybe one day I will write a new blogpost about the ``post-Ph.D.’’ life, but now it’s not the time as I have just finished it at the time of the writing. Thus in this last paragraph, I’ll quickly describe some considerations about this phase.

In the end I went for industry. But from my understanding all roads are open, even though for an academic path you should publish a lot. I opted for industry because, of course, money but also because I wanted to do something more applied into real-world as well as I didn’t like some aspects of academia. For now, I’m joining Qualcomm to work on vulnerability stuff even though I did interviews with other companies before choosing. A consideration related to this is the application time: if you plan to stay in industry, do not apply too early neither too late. I think you can start to search 4-5 months before the defense, but be aware that recruiters very often know that a Ph.D. thesis is a delicate period for an individual, and may prefer other more immediate applications. Also, positions after a Ph.D. are super-specialized (unless you want to do the consultant), bear with it. It may take some time before finding a job that matches all your requirements. But on the other hand, you can definitely start with a job and change it in a second moment when you find a better one.

My final comment? Start a Ph.D. only if you feel this is the right road for you. Then it will become one of the best periods (and jobs) of your life. Don’t get confused by people who say this is needed for working or other similar things. The risk to trash your time is very high in this second case.

I hope that with these words I clarified some aspects/gave some hints about what doing a Ph.D. is. Do you have more questions? Drop me a message on direct (twitter) or an email. Eventually I also plan to add other paragraphs to this list, but let’s see.