{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Running simulations through the cloud\n",
"\n",
"This notebook is a tutorial of the API used for submitting simulations\n",
"to our servers."
],
"id": "16fe2986-9810-4def-afea-f3fd0630b45d"
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import tidy3d as td\n",
"import tidy3d.web as web\n",
"\n",
"# set logging level to ERROR because we'll only running simulations to demonstrate the web API, we don't care about the results.\n",
"td.config.logging.level = \"ERROR\""
],
"id": "f9fc8584"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Let’s set up a simple simulation to get started."
],
"id": "218c4371-af34-4049-96d1-819e03c3ea2a"
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# whether to print output in web functions, note: if False (default) they will all run silently\n",
"verbose = True\n",
"\n",
"# set up parameters of simulation\n",
"dl = 0.05\n",
"pml = td.PML()\n",
"sim_size = [4, 4, 4]\n",
"freq0 = 3e14\n",
"fwidth = 1e13\n",
"run_time = 1 / fwidth\n",
"\n",
"# create structure\n",
"dielectric = td.Medium.from_nk(n=2, k=0, freq=freq0)\n",
"square = td.Structure(geometry=td.Box(center=[0, 0, 0], size=[1.5, 1.5, 1.5]), medium=dielectric)\n",
"\n",
"# create source\n",
"source = td.UniformCurrentSource(\n",
" center=(-1.5, 0, 0),\n",
" size=(0, 0.4, 0.4),\n",
" source_time=td.GaussianPulse(freq0=freq0, fwidth=fwidth),\n",
" polarization=\"Ex\",\n",
")\n",
"\n",
"# create monitor\n",
"monitor = td.FieldMonitor(\n",
" fields=[\"Ex\", \"Ey\", \"Ez\"],\n",
" center=(0, 0, 0),\n",
" size=(td.inf, td.inf, 0),\n",
" freqs=[freq0],\n",
" name=\"field\",\n",
")\n",
"\n",
"# Initialize simulation\n",
"sim = td.Simulation(\n",
" size=sim_size,\n",
" grid_spec=td.GridSpec.uniform(dl),\n",
" structures=[square],\n",
" sources=[source],\n",
" monitors=[monitor],\n",
" run_time=run_time,\n",
" boundary_spec=td.BoundarySpec.all_sides(boundary=pml),\n",
")"
],
"id": "ec0ba676"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running simulation manually\n",
"\n",
"For the most control, you can run the simulation through the Tidy3D web\n",
"API. Each simulation running on the server is identified by a `task_id`,\n",
"which must be specified in the web API. Let’s walk through submitting a\n",
"simulation this way."
],
"id": "a0300e47-54fa-475a-8386-87e8cac9eb5f"
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:14:24 UTC Created task 'webAPI' with resource_id \n",
" 'fdve-974072f6-12bb-41c3-b3ad-5f8fd2711af0' and task_type 'FDTD'. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" View task using web UI at \n",
" 'https://tidy3d.simulation.cloud/workbench?taskId=fdve-974072f6-12b\n",
" b-41c3-b3ad-5f8fd2711af0'. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" Task folder: 'default'. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:14:26 UTC Estimated FlexCredit cost: 0.025. This assumes the FDTD solver runs\n",
" for the full simulation time; if early shutoff is reached, the \n",
" billed cost can be lower. Use 'web.real_cost(task_id)' to get the \n",
" billed FlexCredit cost after a simulation run. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:14:27 UTC status = queued \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" To cancel the simulation, use 'web.abort(task_id)' or \n",
" 'web.delete(task_id)' or abort/delete the task in the web UI. \n",
" Terminating the Python script will not stop the job running on the \n",
" cloud. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:15:48 UTC starting up solver \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" running solver \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:15:49 UTC status = success \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" View simulation result at \n",
" 'https://tidy3d.simulation.cloud/workbench?taskId=fdve-974072f6-12b\n",
" b-41c3-b3ad-5f8fd2711af0'. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:15:50 UTC Loading results from data/sim.hdf5 \n",
""
]
}
}
],
"source": [
"# upload the simulation to our servers\n",
"task_id = web.upload(sim, task_name=\"webAPI\", verbose=verbose)\n",
"\n",
"# start the simulation running\n",
"web.start(task_id)\n",
"\n",
"# monitor the simulation, don't move on to next line until completed.\n",
"web.monitor(task_id, verbose=verbose)\n",
"\n",
"# download the results and load into a simulation data object for plotting, post processing etc.\n",
"sim_data = web.load(task_id, path=\"data/sim.hdf5\", verbose=verbose)"
],
"id": "f0e71d59"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While we broke down each of the individual steps above, one can also\n",
"perform the entire process in one line by calling the\n",
"[web.run()](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.run.run.html)\n",
"function as follows.\n",
"\n",
"``` python\n",
"sim_data = web.run(sim, task_name='webAPI', path='data/sim.hdf5')\n",
"```\n",
"\n",
"(We won’t run it again in this notebook to save time).\n",
"\n",
"Sometimes this is more convenient, but other times it can be helpful to\n",
"have the steps broken down one by one, for example if the simulation is\n",
"long, you may want to\n",
"[web.start](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.webapi.start.html)\n",
"and then exit the session and load the results at a later time using\n",
"[web.load](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.webapi.load.html).\n",
"\n",
"## Job Container\n",
"\n",
"The\n",
"[web.Job](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.container.Job.html)\n",
"interface provides a more convenient way to manage single simulations,\n",
"mainly because it eliminates the need for keeping track of the `task_id`\n",
"and original\n",
"[Simulation](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.Simulation.html).\n",
"\n",
"We can get the cost estimate of running the task before actually running\n",
"it. This prevents us from accidentally running large jobs that we set up\n",
"by mistake. The estimated cost is the maximum cost corresponding to\n",
"running all the time steps."
],
"id": "2ed49347-8c97-4a03-a28e-0ad314e12ae4"
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:15:51 UTC Loading simulation from local cache. View cached task using web UI \n",
" at \n",
" 'https://tidy3d.simulation.cloud/workbench?taskId=fdve-974072f6-12b\n",
" b-41c3-b3ad-5f8fd2711af0'. \n",
""
]
}
},
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
"08:15:52 UTC Estimated FlexCredit cost: 0.025. This assumes the FDTD solver runs\n",
" for the full simulation time; if early shutoff is reached, the \n",
" billed cost can be lower. Use 'web.real_cost(task_id)' to get the \n",
" billed FlexCredit cost after a simulation run. \n",
""
]
}
}
],
"source": [
"# initializes job, puts task on server (but doesn't run it)\n",
"job = web.Job(simulation=sim, task_name=\"job\", verbose=verbose)\n",
"\n",
"# estimate the maximum cost\n",
"estimated_cost = web.estimate_cost(job.task_id)"
],
"id": "7fa41049"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"While\n",
"[Job](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.container.Job.html)\n",
"has methods with names identical to the web API functions above, which\n",
"give some more granular control, it is often most convenient to call\n",
"`.run()` so we will do that now."
],
"id": "555ee50e-81ec-4f98-9797-b85dc036d605"
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"# start job, monitor, and load results\n",
"sim_data = job.run(path=\"data/sim.hdf5\")"
],
"id": "d61ca030"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Another convenient thing about\n",
"[Job](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.container.Job.html)\n",
"objects is that they can be saved and loaded just like other Tidy3d\n",
"components.\n",
"\n",
"This is convenient if you want to save and load up the results of a job\n",
"that has already finished, without needing to know the `task_id`."
],
"id": "2373aee3-a72b-4847-aa8e-a20e413e3a93"
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" Loading simulation from local cache. View cached task using web UI \n",
" at \n",
" 'https://tidy3d.simulation.cloud/workbench?taskId=fdve-974072f6-12b\n",
" b-41c3-b3ad-5f8fd2711af0'. \n",
""
]
}
}
],
"source": [
"# saves the job metadata to a single file\n",
"job.to_file(\"data/job.json\")\n",
"\n",
"# can exit session, break here, or continue in new session.\n",
"\n",
"# load the job metadata from file\n",
"job_loaded = web.Job.from_file(\"data/job.json\")\n",
"\n",
"# download the data from the server and load it into a SimulationData object.\n",
"sim_data = job_loaded.load(path=\"data/sim.hdf5\")"
],
"id": "b1b1a100"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running multiple simulations with `web.run`\n",
"\n",
"Commonly one needs to submit multiple simulations.\n",
"\n",
"As of version `2.10`,\n",
"[web.run](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.web.api.run.run.html)\n",
"can directly submit and monitor any nested combination of dictionaries,\n",
"lists, and tuples of simulations.\n",
"\n",
"This makes it a convenient replacement for batch-style workflows while\n",
"preserving the input structure in the returned results."
],
"id": "7ef488f3-8973-4535-b7ac-7c94ab58f37b"
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"output_type": "display_data",
"metadata": {},
"data": {
"text/html": [
" Loading simulation from local cache. View cached task using web UI \n",
" at \n",
" 'https://tidy3d.simulation.cloud/workbench?taskId=fdve-974072f6-12b\n",
" b-41c3-b3ad-5f8fd2711af0'. \n",
""
]
}
}
],
"source": [
"# make a dictionary of {task name: simulation} for demonstration\n",
"sims = {f\"sim_{i}\": sim for i in range(3)}\n",
"\n",
"# run all simulations and return a dictionary with the same task names\n",
"batch_results = web.run(sims, verbose=verbose)"
],
"id": "936d3252"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The returned object preserves the input structure. Since we used a\n",
"dictionary of simulations, the result is also a dictionary keyed by task\n",
"name.\n",
"\n",
"Each value is a\n",
"[SimulationData](https://docs.flexcompute.com/projects/tidy3d/en/latest/api/_autosummary/tidy3d.SimulationData.html)\n",
"object, so the results can be indexed directly with\n",
"`batch_results[task_name]` or iterated with `batch_results.items()`.\n",
"\n",
"This makes it straightforward to organize parameter scans and other\n",
"grouped studies without explicitly creating a `Batch` container."
],
"id": "2bf8f2e4-683d-455d-94fd-094d38964a71"
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"{'sim_0': 6473211.0, 'sim_1': 6473211.0, 'sim_2': 6473211.0}"
]
}
],
"source": [
"# grab the sum of intensities in the simulation one by one (to save memory)\n",
"intensities = {}\n",
"for task_name, sim_data in batch_results.items():\n",
" intensity = sim_data.get_intensity(\"field\").sel(f=freq0)\n",
" sum_intensity = float(intensity.sum((\"x\", \"y\")).values[0])\n",
" intensities[task_name] = sum_intensity\n",
"\n",
"print(intensities)"
],
"id": "a8536a58"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After going through this tutorial, you have learned how to run\n",
"simulations with Tidy3D via the web API. If you are new to Tidy3D or the\n",
"finite-difference time-domain (FDTD) method, we highly recommend going\n",
"through our [FDTD101](https://www.flexcompute.com/fdtd101/) tutorials\n",
"and [example library](https://www.flexcompute.com/tidy3d/examples/)\n",
"before starting your own simulation adventure."
],
"id": "094e1cdc-2f33-4428-bdf3-feb623d12afc"
}
],
"nbformat": 4,
"nbformat_minor": 5,
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
}
}
}