{ "cells": [ { "cell_type": "markdown", "id": "f3f76509-3b83-49f3-9946-e24b92f6ee8c", "metadata": {}, "source": [ "# MIMIC III (Infinite Horizon)\n", "\n", "In this notebook, we conducted analysis on the MIMIC III data with infinite horizon. We first analyzed the mediation effect and then evaluate the policy of interest and calculated the optimal policy. As informed by the causal structure learning, here we consider Glucose and PaO2_FiO2 as confounders/states, IV_Input as the action, SOFA as the mediator. " ] }, { "cell_type": "code", "execution_count": 1, "id": "a53ca1bf-dae9-4a01-8457-cef602459750", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
icustayidblocGlucosePaO2_FiO2IV_InputSOFAnext_Glucosenext_PaO2_FiO2Died_within_48H
6821006191.0206.0000000891.0206.0000001
6831006391.0206.0000000891.0206.0000001
68410066175.0100.17391313175.0100.1739131
68510067175.096.000000110175.096.0000001
68610068175.096.00000019175.096.0000001
687100610144.0187.234036112144.0187.2340360
\n", "
" ], "text/plain": [ " icustayid bloc Glucose PaO2_FiO2 IV_Input SOFA next_Glucose \\\n", "682 1006 1 91.0 206.000000 0 8 91.0 \n", "683 1006 3 91.0 206.000000 0 8 91.0 \n", "684 1006 6 175.0 100.173913 1 3 175.0 \n", "685 1006 7 175.0 96.000000 1 10 175.0 \n", "686 1006 8 175.0 96.000000 1 9 175.0 \n", "687 1006 10 144.0 187.234036 1 12 144.0 \n", "\n", " next_PaO2_FiO2 Died_within_48H \n", "682 206.000000 1 \n", "683 206.000000 1 \n", "684 100.173913 1 \n", "685 96.000000 1 \n", "686 96.000000 1 \n", "687 187.234036 0 " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import numpy as np\n", "import pandas as pd\n", "import pickle\n", "file = open('mimic3_MRL_data_dict_V2.pickle', 'rb')\n", "mimic3_MRL = pickle.load(file)\n", "mimic3_MRL['reward'] = [1 if r == 0 else r for r in mimic3_MRL['reward']]\n", "mimic3_MRL['reward'] = [0 if r == -1 else r for r in mimic3_MRL['reward']]\n", "MRL_df = pd.read_csv('mimic3_MRL_df_V2.csv')\n", "MRL_df.iloc[np.where(MRL_df['Died_within_48H']==0)[0],-1]=1\n", "MRL_df.iloc[np.where(MRL_df['Died_within_48H']==-1)[0],-1]=0\n", "MRL_df[MRL_df.icustayid==1006]" ] }, { "cell_type": "markdown", "id": "7bdfd9ef-9bee-42bc-8649-f2cc20806ce3", "metadata": {}, "source": [ "## CEL: Mediation Analysis with Infinite Horizon" ] }, { "cell_type": "markdown", "id": "0a161516-9900-4523-bf04-d6f6edd31c76", "metadata": {}, "source": [ "We processed the MIMIC III data similarly to literature on reinforcement learning by setting the reward of each stage prior to the final stage to 0, and the reward of the final stage to the observed value of Died within 48H. In this section, we analyze the average treatment effect (ATE) of a target policy that provides IV input all of the time compared to a control policy that provides no IV input at all. Using the multiply-robust estimator proposed in [1], we decomposed the ATE into four components, including immediate nature dierct effect (INDE), Immediate nature mediator effect (INME), delayed nature direct effect (DNDE), and delayed nature mediator effect (NDDNME), and estimated each of the effect component. The estimation results are summarized in the table below.\n", "\n", "| INDE | INME | DNDE | DNME | ATE |\n", "|---------------|-----|---------------|---------------|---------------|\n", "| -.0181(.0059) | .0061(.0021) | -.0049(.0028) | -.0002(.0011) | -.0171(.0058) |\n", "\n", "Specifically, the ATE of the target policy is significantly negative, with an effect size of .0171. Diving deep, we find that the DNME and DNDE are insignificant, whereas the INDE and INME are all statistically significant. Further, taking the effect size into account, we can conclude that the majority of the average treatment effect is directly due to the actions derived from the target treatment policy, while the part of the effect that can be attributed to the mediators is negligible." ] }, { "cell_type": "code", "execution_count": 2, "id": "722a2e3b-d45c-4035-8033-ff410aef15f9", "metadata": {}, "outputs": [], "source": [ "from causaldm.learners.CEL.MA import ME_MDP" ] }, { "cell_type": "code", "execution_count": 3, "id": "e35283e0-6646-4f14-ab56-ab7afabc7740", "metadata": {}, "outputs": [], "source": [ "# Control Policy\n", "def control_policy(state = None, dim_state=None, action=None, get_a = False):\n", " # fixed policy with fixed action 0\n", " if get_a:\n", " action_value = np.array([0])\n", " else:\n", " state = np.copy(state).reshape(-1,dim_state)\n", " NT = state.shape[0]\n", " if action is None:\n", " action_value = np.array([0]*NT)\n", " else:\n", " action = np.copy(action).flatten()\n", " if len(action) == 1 and NT>1:\n", " action = action * np.ones(NT)\n", " action_value = 1-action\n", " return action_value\n", "def target_policy(state, dim_state = 1, action=None):\n", " state = np.copy(state).reshape((-1, dim_state))\n", " NT = state.shape[0]\n", " pa = 1 * np.ones(NT)\n", " if action is None:\n", " if NT == 1:\n", " pa = pa[0]\n", " prob_arr = np.array([1-pa, pa])\n", " action_value = np.random.choice([0, 1], 1, p=prob_arr)\n", " else:\n", " raise ValueError('No random for matrix input')\n", " else:\n", " action = np.copy(action).flatten()\n", " action_value = pa * action + (1-pa) * (1-action)\n", " return action_value" ] }, { "cell_type": "code", "execution_count": 4, "id": "4a88c374-41a3-4269-997f-c0e87ce586f3", "metadata": {}, "outputs": [], "source": [ "#Fixed hyper-parameter--no need to modify\n", "MCMC = 50\n", "truncate = 50\n", "problearner_parameters = {\"splitter\":[\"best\",\"random\"], \"max_depth\" : range(1,50)},\n", "dim_state=2; dim_mediator = 1\n", "ratio_ndim = 10\n", "d = 2\n", "L = 5\n", "scaler = 'Identity'\n", "method = \"Robust\"\n", "seed = 0\n", "r_model = \"OLS\"\n", "Q_settings = {'scaler': 'Identity','product_tensor': False, 'beta': 3/7, \n", " 'include_intercept': False, \n", " 'penalty': 10**(-4),'d': d, 'min_L': L, 't_dependent_Q': False}" ] }, { "cell_type": "code", "execution_count": 5, "id": "0daebafc-3006-46e5-905d-1f2a3f1fd3c9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Building 0-th basis spline (total 3 state-mediator dimemsion) which has 3 basis, in total 3 features \n", "Building 1-th basis spline (total 3 state-mediator dimemsion) which has 3 basis, in total 6 features \n", "Building 2-th basis spline (total 3 state-mediator dimemsion) which has 3 basis, in total 9 features \n" ] }, { "data": { "text/plain": [ "(-0.01810020554867914,\n", " 0.006066387156687134,\n", " -0.004866328022066192,\n", " -0.00018158807389352197,\n", " -0.017081734487951722)" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Robust_est = ME_MDP.evaluator(mimic3_MRL, r_model = r_model,\n", " problearner_parameters = problearner_parameters,\n", " ratio_ndim = ratio_ndim, truncate = truncate, l2penalty = 10**(-4),\n", " target_policy=target_policy, control_policy = control_policy, \n", " dim_state = dim_state, dim_mediator = dim_mediator, \n", " Q_settings = Q_settings,\n", " MCMC = MCMC,\n", " seed = seed, nature_decomp = True, method = method)\n", "\n", "Robust_est.estimate_DE_ME()\n", "Robust_est.est_IDE, Robust_est.IME, Robust_est.DDE, Robust_est.DME, Robust_est.TE" ] }, { "cell_type": "code", "execution_count": 6, "id": "dfff7a47-7449-4255-a121-e54b1dfc4a08", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(0.0058689011135968135,\n", " 0.002110278954293422,\n", " 0.002770561709572756,\n", " 0.0010678186846614852,\n", " 0.005821662648181514)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Robust_est.IDE_se, Robust_est.IME_se, Robust_est.DDE_se, Robust_est.DME_se, Robust_est.TE_se" ] }, { "cell_type": "markdown", "id": "87dcd03a-13db-4924-8340-3df258236487", "metadata": {}, "source": [ "## Reference\n", "\n", "[1] Ge, L., Wang, J., Shi, C., Wu, Z., & Song, R. (2023). A Reinforcement Learning Framework for Dynamic Mediation Analysis. arXiv preprint arXiv:2301.13348." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.12" } }, "nbformat": 4, "nbformat_minor": 5 }