Skip to content
Fabio Serra Pereira

← Map

FlowFactory

2024 · TU Darmstadt · master’s thesis

A factory simulated with Petri nets. A Q-learning agent decides when something fails.

Simulation

  • Petri nets
  • Learning
  • Decision under failure
lights up: decide

The fault

The rule waits. The agent goes around.

A fault in the Petri net: the fixed rule waits, the agent goes aroundA lightning bolt hits the fourth place in a line of six, and the fault switches between stock, machine and labor. The gray token, the fixed rule, stops there; the orange token, the agent, goes around below, through the ready part, and reaches the flag.stockmachinelaborready part A fault in the Petri net: the fixed rule waits, the agent goes aroundA lightning bolt hits the fourth place in a line of six, and the fault switches between stock, machine and labor. The gray token, the fixed rule, stops there; the orange token, the agent, goes around below, through the ready part, and reaches the flag.stockmachinelaborready part
  • fixed rule (Petri net)
  • agent (Q-learning)

The agent decides:

  • the order of the jobs
  • the route of each part
  • when to order material

See the original factory ↗See the original Petri net ↗

Illustration. The thesis’s three faults, one at a time.

The result

In the simulation, production ends earlier.

Date on which the simulated production ends, in each scenarioIn every scenario the agent finishes before the fixed rule. No fault, 11 Dec against 14 Dec; stock, 14 Dec against 18 Dec; machine, 13 Dec against 5 Jan; labor, 13 Dec against 15 Jan.15 Jul1 Dec15 Dec1 Jan15 Janno fault11 Dec14 Decstock14 Dec18 Decmachine13 Dec5 Janlabor13 Dec15 Jan Date on which the simulated production ends, in each scenarioIn every scenario the agent finishes before the fixed rule. No fault, 11 Dec against 14 Dec; stock, 14 Dec against 18 Dec; machine, 13 Dec against 5 Jan; labor, 13 Dec against 15 Jan.1 Dec15 Dec1 Jan15 Janno fault11 Dec14 Decstock14 Dec18 Decmachine13 Dec5 Janlabor13 Dec15 Jan
  • agent (Q-learning)
  • fixed rule (Petri net)
All simulations start on 15 July 2024. Date on which production ends.

The price

Deciding this way costs hours of computing.

Computing time in each scenario: agent and fixed ruleThe agent takes 217.99, 534.42, 262.28 and 267.67 minutes to compute the four scenarios. The fixed rule takes 1.72, 1.76, 1.72 and 1.61 minutes.0100200300400500minno fault217.99 min1.72 minstock534.42* min1.76 minmachine262.28 min1.72 minlabor267.67 min1.61 min Computing time in each scenario: agent and fixed ruleThe agent takes 217.99, 534.42, 262.28 and 267.67 minutes to compute the four scenarios. The fixed rule takes 1.72, 1.76, 1.72 and 1.61 minutes.0100200300400500minno fault217.99 min1.72 minstock534.42* min1.76 minmachine262.28 min1.72 minlabor267.67 min1.61 min
  • agent (Q-learning)
  • fixed rule (Petri net)
Computing time for each scenario, on the same computer. * probably with other programs open.

Still open

  • It is a simulation, with times synthesized from the literature.
  • The agent knows the jobs in advance. Real time comes later.
  • The thesis suggests SARSA and DQN. DQN came back in Curtailment.

Tools

  • Q-learning
  • Markov decision process
  • Petri nets
  • MATLAB R2024a
  • Reinforcement Learning Toolbox
  • V-model
  • makespan

Document

Development of a resilient Reinforcement Learning-based decision algorithm for order scheduling · master’s thesis · TU Darmstadt, PTW, FlowFactory Lab · 2024

Next station

Next stationDigital ShadowA real process with a model of itself. ← Sustainability.data