{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1eb0ae4a",
   "metadata": {},
   "source": [
    "# Data access\n",
    "\n",
    "IceCube has a bunch of public datasets available at [https://icecube.wisc.edu/science/data-releases/](https://icecube.wisc.edu/science/data-releases/). `icecube_tools` provides an easy interface to this repository so that you can download and organise your data through python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0dbbba1a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:31.811040Z",
     "iopub.status.busy": "2024-03-22T09:55:31.810817Z",
     "iopub.status.idle": "2024-03-22T09:55:32.617616Z",
     "shell.execute_reply": "2024-03-22T09:55:32.616853Z"
    }
   },
   "outputs": [],
   "source": [
    "from icecube_tools.utils.data import IceCubeData"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "165784dc",
   "metadata": {},
   "source": [
    "The `IceCubeData` class provides this functionality. Upon initialisation, `IceCubeData` queries the website using HTTP requests to check what datasets are currently available. By default, this request is cached to avoid spamming the IceCube website. However, you can use the keyword argument `update` to override this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "42c00d09",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:32.621082Z",
     "iopub.status.busy": "2024-03-22T09:55:32.620563Z",
     "iopub.status.idle": "2024-03-22T09:55:33.122030Z",
     "shell.execute_reply": "2024-03-22T09:55:33.121312Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2021_03-all_data_releases.zip',\n",
       " '20080911_AMANDA_7_Year_Data.zip',\n",
       " '20090521_IceCube-22_Solar_WIMP_Data.zip',\n",
       " '20110905_IceCube-40_String_Data.zip',\n",
       " '20131121_Search_for_contained_neutrino_events_at_energies_above_30_TeV_in_2_years_of_data.zip',\n",
       " '20150127_IceCube_Oscillations%20_3_years_muon_neutrino_disappearance_data.zip',\n",
       " '20150219_Search_for_contained_neutrino_events_at_energies_greater_than_1_TeV_in_2_years_of_data.zip',\n",
       " '20150619_IceCube-59%20_Search_for_point_sources_using_muon_events.zip',\n",
       " '20150820_Astrophysical_muon_neutrino_flux_in_the_northern_sky_with_2_years_of_IceCube_data.zip',\n",
       " '20151021_Observation_of_Astrophysical_Neutrinos_in_Four_Years_of_IceCube_Data.zip',\n",
       " '20160105_The_79-string_IceCube_search_for_dark_matter.zip',\n",
       " '20160624_Search_for_sterile_neutrinos_with_one_year_of_IceCube_data.zip',\n",
       " '20161101_Search_for_point_sources_with_first_year_of_IC86_data.zip',\n",
       " '20161115_A_combined_maximum-likelihood_analysis_of_the_astrophysical_neutrino_flux.zip',\n",
       " '20180213_Measurement_of_atmospheric_neutrino_oscillations_with_three_years_of_data_from_the_full_sky.zip',\n",
       " '20180712_IceCube_catalog_of_alert_events_up_through_IceCube-170922A.zip',\n",
       " '20180712_IceCube_data_from_2008_to_2017_related_to_analysis_of_TXS_0506+056.zip',\n",
       " '20181018_All-sky_point-source_IceCube_data%20_years_2010-2012.zip',\n",
       " '20190515_Three-year_high-statistics_neutrino_oscillation_samples.zip',\n",
       " '20190904_Bayesian_posterior_for_IceCube_7-year_point-source_search_with_neutrino-count_statistics.zip',\n",
       " '20200227_All-sky_point-source_IceCube_data%20_years_2012-2015.zip',\n",
       " '20200421_IceCube_Upgrade_Neutrino_Monte_Carlo_Simulation.zip',\n",
       " '20200514_South_Pole_ice_temperature.zip',\n",
       " '20210126_PS-IC40-IC86_VII.zip',\n",
       " '20210310_IceCube_data_for_the_first_Glashow_resonance_candidate.zip',\n",
       " '20211217_HESE-7-5-year-data.zip',\n",
       " '20220201_Density_of_GeV_muons_in_air_showers_measured_with_IceTop.zip',\n",
       " '20220902_Evidence_for_neutrino_emission_from_the_nearby_active_galaxy_NGC_1068_data.zip',\n",
       " '20220913_Evidence_for_neutrino_emission_from_the_nearby_active_galaxy_NGC_1068_data.zip',\n",
       " '20221028_Observation_of_High-Energy_Neutrinos_from_the_Galactic_Plane.zip',\n",
       " '20221208_Observation_of_High-Energy_Neutrinos_from_the_Galactic_Plane.zip',\n",
       " '20230424_Observation_of_High-Energy_Neutrinos_from_the_Galactic_Plane.zip',\n",
       " 'ic22-solar-wimp-histograms.zip']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data = IceCubeData(update=True)\n",
    "\n",
    "# The available datasets\n",
    "my_data.datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e09f8869",
   "metadata": {},
   "source": [
    "You can use the `find` method to pick out datasets you are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ae728eb3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:33.124650Z",
     "iopub.status.busy": "2024-03-22T09:55:33.124261Z",
     "iopub.status.idle": "2024-03-22T09:55:33.128413Z",
     "shell.execute_reply": "2024-03-22T09:55:33.127886Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['20181018_All-sky_point-source_IceCube_data%20_years_2010-2012.zip']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "found_dataset = my_data.find(\"20181018\")\n",
    "found_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd0cc71c",
   "metadata": {},
   "source": [
    "The `my_data` object has been inititalised to store data in the package's default location (\"~/.icecube_data\"). This is where other `icecube_tools` modules will look for stored data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "528be44d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:33.130747Z",
     "iopub.status.busy": "2024-03-22T09:55:33.130371Z",
     "iopub.status.idle": "2024-03-22T09:55:33.134369Z",
     "shell.execute_reply": "2024-03-22T09:55:33.133829Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/runner/.icecube_data'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.data_directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b005fdd",
   "metadata": {},
   "source": [
    "The `fetch` method will download this dataset to this default location for later use by `icecube_tools`. This method takes a list of names, so can also be used to download multiple datasets. `fetch` has a built in delay of a few seconds between HTTP requests to avoid spamming the website. `fetch` will not overwrite files by default, but this can be forced with `overwrite=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c3041dd2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:33.136762Z",
     "iopub.status.busy": "2024-03-22T09:55:33.136398Z",
     "iopub.status.idle": "2024-03-22T09:55:45.697887Z",
     "shell.execute_reply": "2024-03-22T09:55:45.697182Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:   0%|          | 0/11730609 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:   9%|▉         | 1048576/11730609 [00:00<00:09, 1070870.92it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  18%|█▊        | 2097152/11730609 [00:01<00:05, 1812901.08it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  27%|██▋       | 3145728/11730609 [00:01<00:03, 2730729.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  36%|███▌      | 4194304/11730609 [00:01<00:02, 3660828.45it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  54%|█████▎    | 6291456/11730609 [00:01<00:00, 5972742.93it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  72%|███████▏  | 8388608/11730609 [00:01<00:00, 8041849.70it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  98%|█████████▊| 11534336/11730609 [00:02<00:00, 11332662.63it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...: 100%|██████████| 11730609/11730609 [00:02<00:00, 5811211.78it/s] "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "my_data.fetch(found_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74cfd7c1",
   "metadata": {},
   "source": [
    "You may not want to use `icecube_tools` for other stuff, so you can also fetch to a specificed location with the keyword `write_to`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6be53097",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:45.701076Z",
     "iopub.status.busy": "2024-03-22T09:55:45.700653Z",
     "iopub.status.idle": "2024-03-22T09:55:55.543650Z",
     "shell.execute_reply": "2024-03-22T09:55:55.542926Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:   0%|          | 0/11730609 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:   9%|▉         | 1048576/11730609 [00:01<00:11, 940985.25it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  18%|█▊        | 2097152/11730609 [00:01<00:05, 1752336.33it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  27%|██▋       | 3145728/11730609 [00:01<00:03, 2459346.64it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  45%|████▍     | 5242880/11730609 [00:01<00:01, 4174728.68it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  63%|██████▎   | 7340032/11730609 [00:01<00:00, 6072432.45it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...:  80%|████████  | 9437184/11730609 [00:02<00:00, 7979942.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018_All-sky_point-source_IceCube_da...: 100%|██████████| 11730609/11730609 [00:02<00:00, 5410751.14it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "my_data.fetch(found_dataset, write_to=\"data\", overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06684c45",
   "metadata": {},
   "source": [
    "For convenience, there is also the `fetch_all_to` method to download all the available data to a specified location. We comment this here as it can take a while to execute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b09f792e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-22T09:55:55.546482Z",
     "iopub.status.busy": "2024-03-22T09:55:55.546277Z",
     "iopub.status.idle": "2024-03-22T09:55:55.549305Z",
     "shell.execute_reply": "2024-03-22T09:55:55.548770Z"
    }
   },
   "outputs": [],
   "source": [
    "# my_data.fetch_all_to(\"data\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "icecube_dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}