PhD Position Calls Archive
2018: Sequential Learning in Dynamic Environments
We are seeking candidates for the following position at SequeL/Inria Lille. We are in the process of interviewing the candidates already, so please get in touch with us as early as possible.
Sequential Learning in Dynamic Environments, SequeL team, Inria, Lille, France (PhD position)
Keywords: multi-arm bandits, stochastic optimization, reinforcement learning, monte-carlo tree search, planning, changing and non-stationary environments
This PhD program is focused on sequential learning in structured and dynamic environments. The key aspect of this problem is that relatively little knowledge of the environment is available beforehand, and the learner (a virtual or physical agent) has to sequentially interact with environment to learn its structure and then act optimally. This problem encompasses a wide range of applications depending on the definition of the structure of the environment, the sequential nature of the interaction between learner and environment, and the type of dynamics and evolution. In particular, this PhD program will be driven by two application domains of scientific and societal importance: planning in games and in random environments. In both setups, the paradigm of reinforcement learning, in which the learner collects rewards to guide his/her actions, will be useful. In many cases the dynamic of the sequential interaction is particularly complex, since rewards keep changing over time and the set of actions (e.g., items that can be recommended in a recommender systems) continuously evolve (products may be added or removed or change characteristics).
Finally, both applications usually involve large spaces which asks for efficient learning algorithms with minimal time, space, and sample complexity. In general, a successful sequential learning strategy should efficiently allocate the limited resources to exploitation (making the best decision based on our current, but possibly imperfect, knowledge) and to exploration (decisions that may appear sub-optimal but which may reduce the uncertainty and, as a result, could improve the relevance of future decisions).
Background:
The main mathematical models of reference to deal with the problem of sequential learning are multi-armed bandits and reinforcement learning. The multi-armed bandit problem captures the essence of the exploration-exploitation trade-off in an unknown environment where only noisy observations of the performance of a set of actions, called arms (e.g., the preference of a user for a product) are available. The objective is to identify the arm with the best performance. On the other hand, reinforcement learning formalizes the more general problem of decision-making under uncertainty when the state of the environment evolves in response to the action taken by the learner.
Objectives:
Many complex autonomous systems (e.g., electrical distribution networks, or smart grids) repeatedly select actions with the aim of achieving a given objective. Reinforcement learning (RL) offers a powerful framework for acquiring adaptive behavior in this setting, associating a scalar reward with each action and learning from experience which action to select to maximize long-term reward. Although RL has produced impressive results recently (e.g., achieving human-level play in Atari games and beating the human world champion in the board game Go), most existing solutions only work under strong assumptions: the environment model is stationary, the objective is fixed, and trials end once the objective is met.
The aim of this project is to advance the state of the art of fundamental research in lifelong RL by developing several novel RL algorithms that relax the above assumptions. The new algorithms should be robust to environmental changes, both in terms of the observations that the system can make and the actions that the system can perform. Moreover, the algorithms should be able to operate over long periods of time while achieving different objectives. The proposed algorithms will address three key problems related to lifelong RL: planning, exploration, and task decomposition. Planning is the problem of computing an action selection strategy given a (possibly partial) model of the task at hand. Exploration is the problem of selecting actions with the aim of mapping out the environment rather than achieving a particular objective. Task decomposition is the problem of defining different objectives and assigning a separate action selection strategy to each. The algorithms will be evaluated in two realistic scenarios: active network management for electrical distribution networks, and microgrid management. A test protocol will be developed to evaluate each individual algorithm, as well as their combinations.
Job Description:
The PhD candidate will focus on one or more issues related to sequential learning in structured and evolving problems. The PhD candidate will first acquire expertise in different topics of machine learning such as online learning, multi-armed bandit, statistical learning theory, reinforcement learning, approximate dynamic programming, and algorithmic game theory. Then, the PhD candidate is expected to contribute to the advancement of the literature on this problem along many different lines: methodological (e.g., definition of general abstract models for a wide range of decision-making problems), theoretical (e.g., near optimality performance guarantees), and algorithmic (e.g., development of novel algorithms for specific decision-making problems).
The research activity of the PhD candidate will be closely related to EU Chistera Delta project (https://www.chistera.eu/projects/delta). This will allow the PhD candidate to develop collaborations with other researchers participating in this research project and it may also allow him/her to spend part of his research activity at partner laboratories such as Montanuniversitat Leoben (Austria), Universita Pompeu Fabra (Spain) or Université de Liège (Belgium). Possibility of internships in the industry research labs such as: Adobe Research in California or DeepMind in UK. The starting date of PhD program is flexible.
Profile:
The applicant must have a Master of Science in Computer Science, Statistics, Math, or related fields, possibly with background in reinforcement learning, bandits, or optimization. The working language in the lab is English, good written and oral communication skills are required.
Application:
We encourage the applicants to contact us immediately at least with a CV. The full application should include a brief description of research interests and past experience, a CV, degrees and grades, a copy of Master thesis (or a draft thereof), motivation letter (short but pertinent to this call), relevant publications, and other relevant documents. Candidates are encouraged to provide letter(s) of recommendation and contact information to reference persons. Please send your application in one single pdf to emilie.kaufmann-at-univ-lille1.fr and michal.valko-at-inria.fr.
- Applications requested until position is filled
- Duration: 3 years (a full time position)
- Starting date: October 15, 2018 (negotiable)
- Supervisors: Emilie Kaufmann and Michal Valko
- Place: SequeL, Inria Lille - Nord Europe
Working Environment:
The PhD candidate will work at SequeL (https://team.inria.fr/scool/) lab at Inria Lille - Nord Europe located in Lille. Inria (https://www.inria.fr/fr) is France's leading institution in Computer Science, with over 2800 scientists employed, of which around 250 in Lille. Lille is the capital of the north of France, a metropolis with 1 million inhabitants, with excellent train connection to Brussels (30 min), Paris (1h) and London (1h30). The SequeL lab is a dynamic lab at Inria with over 20 researchers (including PhD students) which covers several aspects of machine learning from theory to applications, including statistical learning, reinforcement learning, and sequential learning.
Benefits:
- Duration: 36 months – starting date of the contract: October 2018, 15th (negotiable)
- Salary: €1982 the first two years and €2085 the third year (before taxes, taxes are around €380)
- Possibility of French courses
- Help for housing
- Participation for public transport
- Scientific Resident card and help for husband/wife visa
2014: Efficient Sequential Learning in Structured and Constrained Environments
Efficient Sequential Learning in Structured and Constrained Environments, SequeL team, Inria, Lille, France (PhD position)
Keywords: multi-arm bandit, stochastic optimization, reinforcement learning, apprenticeship learning, learning on graphs, transfer learning
This PhD program is focused on the problem of sequential learning in structured and constrained environments. The key aspect of this problem is that relatively little knowledge of the environment is available beforehand, and the learner (a virtual or physical agent) has to sequentially interact with environment to learn its structure and then act optimally. This problem encompasses a wide range of applications depending on the definition of the structure of the environment, the sequential nature of the interaction between learner and environment, and the type of constraints. In particular, this PhD program will be driven by two application domains of great scientific and commercial importance: recommender systems and crowdsourcing. In recommender systems, the sequential interaction is particularly complex since the user-base keeps changing over time (users subscribe and unsubscribe to the service) and the set of items that can be recommended continuously evolve (products may be added or removed or change characteristics). On the other hand, the environment is highly structured, since users may have similar preferences and items share similar features that could be exploited to improve the quality of the recommendations. In crowdsourcing, users are payed to solve subtasks assigned by an automatic system. The main challenge in this scenario is that users may have very heterogeneous skills and the crowdsourcing campaign has strict, and often contradicting, constraints on resources, such as the money invested in the campaign, the time, the desired accuracy in the solution of the global task, etc. In this case, it is crucial that the learner identifies the best allocation of subtasks to users in order to meet most of the desired constraints. Finally, both applications usually involve large amounts of data (e.g., large user-base) which asks for efficient learning algorithms with minimal time, space, and sample complexity. In general, a successful sequential learning strategy should efficiently allocate the constrained resources to exploitation (making the best decision based on our current, but possibly imperfect, knowledge) and to exploration (decisions that may appear sub-optimal but which may reduce the uncertainty and, as a result, could improve the relevance of future decisions).
Background:
The main mathematical models of reference to deal with the problem of sequential learning are multi-armed bandits and reinforcement learning. The multi-armed bandit problem captures the essence of the exploration-exploitation trade-off in an unknown environment where only noisy observations of the performance of a set of arms (e.g., the preference of a user for a product) are available and the objective is to identify the arm with the best performance. On the other hand, reinforcement learning formalizes the more general problem of decision-making under uncertainty when the state of the environment evolves in response to the action taken by the learner.
Objectives:
The objective of this research program is to answer to a number of fundamental questions in sequential learning in structured and constrained environments:
- Sequential transfer learning from multiple and concurrent sources: It is often the case that similar problems belonging to the same domain should be solved over time (e.g., in crowdsourcing the user-base may be almost the same across different tasks). The objective of sequential transfer learning is to construct general knowledge from past experience and reuse it to improve the learning performance over subsequent tasks. The objective is to develop transfer algorithms for exploration-exploitation problems and understand how effective it could be and under which assumptions, negative transfer can be avoided. Furthermore, we also intend to investigate the scenario of transfer from parallel sources, like in distributed learning problems. In this case, the main constraints are time and communication. In fact, each source acts as a node in a network so that at each point in time the information available at different sources is limited and each source may be constrained in communicating the knowledge learned so far to other nodes.
- Learning on Graphs: Graphs offer a natural representation for the structured problems. The users of a social network, the similarity between the content, the location proximity of the sensors: all these can be represented by a (weighted) graph. A decision-making strategy then often queries either nodes or edges, e.g., readings from a sensor network, product offers to a particular user in a social network, etc. Graph representation however comes with several challenges. A large number of nodes poses significant storage and computational difficulties and efficient approximations need to be often designed. Furthermore, real-world graphs are typically dynamic, with changing number of nodes and connections. Finally, there are cases when even the graph structure itself is unknown and its discovery needs to be a part of an efficient strategy. Summing up, learning in realistic graph needs to account for a structure discovery.
- Sequential apprenticeship learning: In reinforcement learning, it is not always easy to define the desired behavior of an agent through a reward function. For instance, in recommender systems the objective is to maximize the long-term revenue, but other criteria may be of interest such as avoiding suggesting the same item too often, favor novel products, and introduce promotions. These behaviors are desirable but difficult to encode as a reward. Nonetheless, an expert may sequentially provide feedbacks to the strategy currently implemented by the learner over time and even propose corrections. The objective is to integrate the imitation learning and reinforcement learning in a unique learning system that can optimize the reward (e.g., long-term revenue) exploiting the corrections and examples provided by the expert. Another issue that should be considered is that the learner may have a finite budget of requests of feedbacks from the supervisor and should use them when it is more needed.
Job Description:
The PhD candidate will focus on one or more issues related to sequential learning in structured and constrained problems. The PhD candidate will first acquire expertise in different topics of machine learning such as online learning, multi-armed bandit, statistical learning theory, reinforcement learning, approximate dynamic programming, and algorithmic game theory. Then, the PhD candidate is expected to contribute to the advancement of the literature on this problem along many different lines: methodological (e.g., definition of general abstract models for a wide range of decision-making problems), theoretical (e.g., near optimality performance guarantees), and algorithmic (e.g., development of novel algorithms for specific decision-making problems). The research activity of the PhD candidate will be closely related to EU CompLACS project. This will allow the PhD candidate to develop collaborations with other researchers participating in these research projects and it may also allow him/her to spend part of his research activity at partner laboratories such as University College London (UK), Montanuniversitat Leoben (Austria), TU Darmstadt, etc. Possibility of internships in the industry research labs such as: Technicolor Research or INTEL Research in California. The starting date of PhD program flexible after October 15th, 2014.
Profile:
The applicant must have a Master of Science in Computer Science, Statistics, or related fields, possibly with background in reinforcement learning, bandits, or optimization. The working language in the lab is English, a good written and oral communication skills are required.
Application:
The application should include a brief description of research interests and past experience, a CV, degrees and grades, a copy of Master thesis (or a draft thereof), motivation letter (short but pertinent to this call), relevant publications, and other relevant documents. Candidates are encouraged to provide letter(s) of recommendation and contact information to reference persons. Please send your application in one single pdf to michal.valko-at-inria.fr or alessandro.lazaric-at-inria.fr. The deadline for the application is April 15th, 2014, but we encourage the applicants to contact us as soon as possible. The final decision will be communicated in the beginning of June.
- Application closing date: April 15, 2014
- Duration: 3 years (a full time position)
- Starting date: October 15, 2014
- Supervisors: Michal Valko, Alessandro Lazaric, and Rémi Munos
- Place: SequeL, Inria Lille - Nord Europe
Working Environment:
The PhD candidate will work at SequeL (https://team.inria.fr/scool/) lab at Inria Lille - Nord Europe located in Lille. Inria (https://www.inria.fr/fr) is France's leading institution in Computer Science, with over 2800 scientists employed, of which around 250 in Lille. Lille is the capital of the north of France, a metropolis with 1 million inhabitants, with excellent train connection to Brussels (30 min), Paris (1h) and London (1h30). The SequeL lab is a dynamic lab at Inria with over 25 researchers (including PhD students) which covers several aspects of machine learning from theory to applications, including statistical learning, reinforcement learning, and sequential learning.
Benefits:
- Duration: 36 months – starting date of the contract: October 2014, 15th
- Salary: 1957,54 € the first two years and 2058,84 € the third year
- Salary after taxes: around 1597,11€ the 1st two years and 1679,76 € the 3rd year (benefits included).
- Possibility of French courses
- Help for housing
- Participation for public transport
- Scientific Resident card and help for husband/wife visa
References:
2013: Funded PhD position in Sequential Learning with Similarities
Funded PhD position in Sequential Learning with Similarities, SequeL team, Inria, Lille, France
Description:
The goal of this PhD position is to design and analyze efficient algorithms for decision making under uncertainty. Specifically, this position will focus on cases when the possible actions are somehow related. This extra information can be provided in a form of weighted graph or a similarity metric. The aim is to provide provably optimal algorithms that can be applied in large-scale scenarios, such as movie recommendation or social networks. The purpose is to minimize feedback that we need to give the algorithm in order to make it "intelligent". In other words, we want the decision-making algorithms that converge fast to nearly-optimal solutions with minimal feedback (samples). For this purpose, we often need to learn about the environment (explore) at the same time as choosing the currently most promising option (exploit). This requires careful allocation of resources, which could be financial costs, CPU time, or a human effort.
Keywords: machine learning, graph-based learning, exploration-exploitation tradeoff, bandit algorithms, learning with similarities, minimal feedback
Objectives:
The multi-armed bandit problem is a simple model to study the trade-off between exploration and exploitation. While simple enough, it displays most of the issues that arise in a variety of decision-making problems under uncertainty. The following problems are the most pertinent to the proposed research program:
- Contextual bandits: This setting is an extension of the multi-armed bandit problem where the best decision depends on the information (context) that is repeatedly given to the decision maker [S09]. Selecting relevant web advertisement or the news feed recommendations are the most common applications of this setting [LCLS10].
- Bandits on Graphs: In many problems, the similarities between the decisions are provided in the form of a graph that relates the pairs of the decisions (nodes), potentially with a weight. This can refer to connections in the social networks [CKLB12] or more general combinatorial problems [BL12, YM11]. Another use is in the recommender systems where we want to discover user preferences (assumed to be smooth on a given graph) with minimal number of queries.
- Stochastic optimization: The optimization of a noisy objective function is a very general problem whose difficulty strictly depends on the properties of the function itself (e.g., linear [AHR08], Lipschitz [BMSS08], submodular [HK12]) and the space the function is defined on (e.g., finite support, continuous). What other interesting settings are (functions and spaces) and what set of assumptions is really needed to successfully optimize the function are issues currently under investigation.
Job Description:
The PhD candidate will focus on one or more issues related to the problem of learning and decision-making under uncertainty with some similarity information. The PhD candidate will first acquire expertise in different topics of machine learning such as online learning, multi-armed bandit, statistical learning theory, and graph-based learning. The candidate is then expected to contribute to the advancement of the literature on this problem along many different lines: methodological (e.g., definition of general abstract models for a wide range of decision-making problems), theoretical (e.g., near optimality performance guarantees), and algorithmic (e.g., development of novel algorithms for specific decision-making problems). The candidate will work with Michal Valko (https://misovalko.github.io/), Rémi Munos, and other members of the lab. The applicant will also have the opportunity to collaborate with researchers in several countries in Europe and USA.
Requirements:
The successful candidate will have a MSc or equivalent degree in computer science with a strong background in theory or in mathematics. Programming skills will be considered a plus. The working language in the lab is English, a good written and oral communication skills are required.
- Application closing date: April 15, 2013
- Duration: 3 years (a full time position)
- Starting date: October 1st, 2013
- Supervisors: Michal Valko and Rémi Munos
- Place: SequeL, Inria Lille - Nord Europe
About Inria:
SequeL (https://team.inria.fr/scool/) is one of the most dynamic labs at Inria (https://www.inria.fr/fr), with over 25 researchers and PhD students working on both fundamental and practical aspects of sequential learning problems: from statistical learning, through reinforcement learning, to games. Lille is the capital of the north of France, a metropolis with 1 million inhabitants, with excellent train connection to Brussels (30 min), Paris (1h) and London (1h30). Established in 1967, Inria is the only public research body fully dedicated to computational sciences. Combining computer sciences with mathematics, Inria's 3,400 researchers strive to invent the digital technologies of the future. Educated at leading international universities, they creatively integrate basic research with applied research and dedicate themselves to solving real problems, collaborating with the main players in public and private research in France and abroad and transferring the fruits of their work to innovative companies. The researchers at Inria published over 4000 articles a year. They are behind over 270 active patents and 105 start-ups. The 171 project teams are distributed in eight research centers located throughout France.
Benefits:
- Duration: 36 months – starting date of the contract: October 2013, 15th
- Salary: 1957,54 € the first two years and 2058,84 € the third year
- Salary after taxes: around 1597,11€ the 1st two years and 1679,76 € the 3rd year (benefits included).
- Possibility of French courses
- Help for housing
- Participation for public transport
- Scientific Resident card and help for husband/wife visa
References:
For further information please send an email to michal.valko-at-inria.fr as soon as possible.