Skip to content
Snippets Groups Projects
Commit 586329bd authored by Russell, Hannah's avatar Russell, Hannah
Browse files

add ems data and update ipynb files

parent c7eaf01e
No related branches found
No related tags found
No related merge requests found
Showing
with 2726 additions and 1 deletion
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
File added
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
Source diff could not be displayed: it is stored in LFS. Options to address this: view the blob.
This diff is collapsed.
%% Cell type:code id:f6efd717 tags: %% Cell type:code id:f6efd717 tags:
``` python ``` python
import glob import glob
import pandas as pd import pandas as pd
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
import numpy as np import numpy as np
# Get xls files list from a folder
path = 'Elbe'
xls_files = glob.glob(path + "/*.xls")
# Read each CSV file into DataFrame
# This creates a list of dataframes
df_list = (pd.read_excel(file) for file in xls_files)
# Concatenate all DataFrames
turbidity = pd.concat(df_list, ignore_index=True)
turbidity['Messwert'] = turbidity['Messwert'].str.replace(",", ".")
turbidity.drop(turbidity[turbidity.Stromkilometer.str.contains('[Nebenge]', na=False)].index, inplace=True)
Stromkilometer = turbidity['Stromkilometer'].astype(float)
Messwert = turbidity['Messwert'].astype(float)
turbidity
``` ```
%% Output %% Output
Gew�sser Wasserk�rper Messstelle \ Gew�sser Wasserk�rper Messstelle \
0 Elbe Elbe-West Blankenese (Strom-km 634,3) 0 Elbe Elbe-West Blankenese (Strom-km 634,3)
1 Elbe Elbe-West Blankenese (Strom-km 634,3) 1 Elbe Elbe-West Blankenese (Strom-km 634,3)
2 Elbe Elbe-West Blankenese (Strom-km 634,3) 2 Elbe Elbe-West Blankenese (Strom-km 634,3)
3 Elbe Elbe-West Blankenese (Strom-km 634,3) 3 Elbe Elbe-West Blankenese (Strom-km 634,3)
4 Elbe Elbe-West Blankenese (Strom-km 634,3) 4 Elbe Elbe-West Blankenese (Strom-km 634,3)
... ... ... ... ... ... ... ...
77103 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6) 77103 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6)
77104 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6) 77104 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6)
77105 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6) 77105 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6)
77106 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6) 77106 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6)
77107 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6) 77107 Elbe Elbe-1 Zehren, linkes Ufer (Strom-km 89,6)
Stromkilometer Parameter Messwert Einheit \ Stromkilometer Parameter Messwert Einheit \
0 634.3 Trübung (FNU) 30.6 FNU 0 634.3 Trübung (FNU) 30.6 FNU
1 634.3 Trübung (FNU) 39.7 FNU 1 634.3 Trübung (FNU) 39.7 FNU
2 634.3 Trübung (FNU) 37.2 FNU 2 634.3 Trübung (FNU) 37.2 FNU
3 634.3 Trübung (FNU) 34.0 FNU 3 634.3 Trübung (FNU) 34.0 FNU
4 634.3 Trübung (FNU) 35.1 FNU 4 634.3 Trübung (FNU) 35.1 FNU
... ... ... ... ... ... ... ... ... ...
77103 89.6 Trübung (FNU) 9.3 FNU 77103 89.6 Trübung (FNU) 9.3 FNU
77104 89.6 Trübung (FNU) 6.4 FNU 77104 89.6 Trübung (FNU) 6.4 FNU
77105 89.6 Trübung (FNU) 6.6 FNU 77105 89.6 Trübung (FNU) 6.6 FNU
77106 89.6 Trübung (FNU) 6.1 FNU 77106 89.6 Trübung (FNU) 6.1 FNU
77107 89.6 Trübung (FNU) 6.7 FNU 77107 89.6 Trübung (FNU) 6.7 FNU
Messwerttyp Medium Messwertart \ Messwerttyp Medium Messwertart \
0 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 0 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
1 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 1 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
2 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 2 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
3 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 3 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
4 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 4 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
... ... ... ... ... ... ... ...
77103 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 77103 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
77104 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 77104 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
77105 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 77105 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
77106 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 77106 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
77107 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert 77107 quantitativ nachgewiesen Wasser - Gesamtprobe Tagesmittelwert
Messvorgang Datum Bezugsjahr Zeit Datum bis \ Messvorgang Datum Bezugsjahr Zeit Datum bis \
0 kontinuierliche Messungen 02.01.1997 1997 NaN NaN 0 kontinuierliche Messungen 02.01.1997 1997 NaN NaN
1 kontinuierliche Messungen 03.01.1997 1997 NaN NaN 1 kontinuierliche Messungen 03.01.1997 1997 NaN NaN
2 kontinuierliche Messungen 06.01.1997 1997 NaN NaN 2 kontinuierliche Messungen 06.01.1997 1997 NaN NaN
3 kontinuierliche Messungen 07.01.1997 1997 NaN NaN 3 kontinuierliche Messungen 07.01.1997 1997 NaN NaN
4 kontinuierliche Messungen 08.01.1997 1997 NaN NaN 4 kontinuierliche Messungen 08.01.1997 1997 NaN NaN
... ... ... ... ... ... ... ... ... ... ... ...
77103 kontinuierliche Messungen 27.12.2021 2021 NaN NaN 77103 kontinuierliche Messungen 27.12.2021 2021 NaN NaN
77104 kontinuierliche Messungen 28.12.2021 2021 NaN NaN 77104 kontinuierliche Messungen 28.12.2021 2021 NaN NaN
77105 kontinuierliche Messungen 29.12.2021 2021 NaN NaN 77105 kontinuierliche Messungen 29.12.2021 2021 NaN NaN
77106 kontinuierliche Messungen 30.12.2021 2021 NaN NaN 77106 kontinuierliche Messungen 30.12.2021 2021 NaN NaN
77107 kontinuierliche Messungen 31.12.2021 2021 NaN NaN 77107 kontinuierliche Messungen 31.12.2021 2021 NaN NaN
Zeit bis Status Analysemethode Bemerkung (Datenausgabe) \ Zeit bis Status Analysemethode Bemerkung (Datenausgabe) \
0 NaN freigegeben - NaN 0 NaN freigegeben - NaN
1 NaN freigegeben - NaN 1 NaN freigegeben - NaN
2 NaN freigegeben - NaN 2 NaN freigegeben - NaN
3 NaN freigegeben - NaN 3 NaN freigegeben - NaN
4 NaN freigegeben - NaN 4 NaN freigegeben - NaN
... ... ... ... ... ... ... ... ... ...
77103 NaN freigegeben DIN EN ISO 7027 (C2) NaN 77103 NaN freigegeben DIN EN ISO 7027 (C2) NaN
77104 NaN freigegeben DIN EN ISO 7027 (C2) NaN 77104 NaN freigegeben DIN EN ISO 7027 (C2) NaN
77105 NaN freigegeben DIN EN ISO 7027 (C2) NaN 77105 NaN freigegeben DIN EN ISO 7027 (C2) NaN
77106 NaN freigegeben DIN EN ISO 7027 (C2) NaN 77106 NaN freigegeben DIN EN ISO 7027 (C2) NaN
77107 NaN freigegeben DIN EN ISO 7027 (C2) NaN 77107 NaN freigegeben DIN EN ISO 7027 (C2) NaN
zus�tzliche Informationen zus�tzliche Informationen
0 NaN 0 NaN
1 NaN 1 NaN
2 NaN 2 NaN
3 NaN 3 NaN
4 NaN 4 NaN
... ... ... ...
77103 NaN 77103 NaN
77104 NaN 77104 NaN
77105 NaN 77105 NaN
77106 NaN 77106 NaN
77107 NaN 77107 NaN
[64216 rows x 20 columns] [64216 rows x 20 columns]
%% Cell type:markdown id:1acc33ee tags:
## pre-processing elbe Chlorophyll data
%% Cell type:code id:cdee84e5 tags:
``` python
# Get xls files list from a folder
path = 'Elbe'
xls_files = glob.glob(path + "/*.xls")
# Read each CSV file into DataFrame
# This creates a list of dataframes
df_list = (pd.read_excel(file) for file in xls_files)
# Concatenate all DataFrames
turbidity = pd.concat(df_list, ignore_index=True)
turbidity['Messwert'] = turbidity['Messwert'].str.replace(",", ".")
turbidity.drop(turbidity[turbidity.Stromkilometer.str.contains('[Nebenge]', na=False)].index, inplace=True)
Stromkilometer = turbidity['Stromkilometer'].astype(float)
Messwert = turbidity['Messwert'].astype(float)
turbidity
```
%% Cell type:code id:bb4beeaa tags: %% Cell type:code id:bb4beeaa tags:
``` python ``` python
plt.scatter(Stromkilometer, Messwert) plt.scatter(Stromkilometer, Messwert)
plt.gca().invert_xaxis() plt.gca().invert_xaxis()
plt.xlim(586,700) #Geestacht is at ~586 km plt.xlim(586,700) #Geestacht is at ~586 km
plt.title('Elbe-- All Years') plt.title('Elbe-- All Years')
plt.xlabel('Kilometer') plt.xlabel('Kilometer')
plt.ylabel('Turbidity FNU') plt.ylabel('Turbidity FNU')
``` ```
%% Output %% Output
Text(0, 0.5, 'Turbidity FNU') Text(0, 0.5, 'Turbidity FNU')
%% Cell type:code id:b638e47d tags: %% Cell type:code id:b638e47d tags:
``` python ``` python
turb_avg = turbidity[['Stromkilometer', 'Messwert']] turb_avg = turbidity[['Stromkilometer', 'Messwert']]
turb_avg.dropna() turb_avg.dropna()
turb_avg = turb_avg.groupby('Stromkilometer', as_index = False).median() #for some reason this is invalid even though median() works... not sure yet what's wrong here turb_avg = turb_avg.groupby('Stromkilometer', as_index = False).median() #for some reason this is invalid even though median() works... not sure yet what's wrong here
turb_avg turb_avg
``` ```
%% Output %% Output
Stromkilometer Messwert Stromkilometer Messwert
0 3.9 10.5 0 3.9 10.5
1 26.5 3.8 1 26.5 3.8
2 89.6 11.7 2 89.6 11.7
3 172.6 14.6 3 172.6 14.6
4 216.6 13.4 4 216.6 13.4
5 318.1 20.0 5 318.1 20.0
6 470.0 10.9 6 470.0 10.9
7 474.5 19.5 7 474.5 19.5
8 609.8 18.2 8 609.8 18.2
9 628.9 30.6 9 628.9 30.6
10 634.3 25.6 10 634.3 25.6
11 660.6 73.8 11 660.6 73.8
%% Cell type:code id:c2739d4d tags: %% Cell type:code id:c2739d4d tags:
``` python ``` python
km_avg = turb_avg['Stromkilometer'] km_avg = turb_avg['Stromkilometer']
Messwert_avg = turb_avg['Messwert'] Messwert_avg = turb_avg['Messwert']
plt.plot(km_avg, Messwert_avg) plt.plot(km_avg, Messwert_avg)
plt.gca().invert_xaxis() plt.gca().invert_xaxis()
plt.title('Elbe-- Median at km') plt.title('Elbe-- Median at km')
plt.xlabel('Kilometer') plt.xlabel('Kilometer')
plt.ylabel('turbidity') plt.ylabel('turbidity')
``` ```
%% Output %% Output
Text(0, 0.5, 'turbidity') Text(0, 0.5, 'turbidity')
%% Cell type:code id:f84bdb72 tags: %% Cell type:code id:f84bdb72 tags:
``` python ``` python
elbe_depth = pd.read_csv('Elbe Depth.csv') elbe_depth = pd.read_csv('Elbe Depth.csv')
elbe_depth['Stromkilometer'] = elbe_depth['Stromkilometer'].astype(float).round(1) elbe_depth['Stromkilometer'] = elbe_depth['Stromkilometer'].astype(float).round(1)
Stromkilometer2 = elbe_depth['Stromkilometer'] Stromkilometer2 = elbe_depth['Stromkilometer']
Depth = elbe_depth['Depth'] Depth = elbe_depth['Depth']
elbe_depth elbe_depth
``` ```
%% Output %% Output
Stromkilometer Depth Stromkilometer Depth
0 537.9 -2.574074 0 537.9 -2.574074
1 546.2 -1.666667 1 546.2 -1.666667
2 561.8 0.131687 2 561.8 0.131687
3 572.3 0.370370 3 572.3 0.370370
4 579.6 1.267490 4 579.6 1.267490
.. ... ... .. ... ...
131 821.8 25.481481 131 821.8 25.481481
132 823.2 26.518519 132 823.2 26.518519
133 824.9 24.888889 133 824.9 24.888889
134 826.0 24.444444 134 826.0 24.444444
135 828.8 25.481481 135 828.8 25.481481
[136 rows x 2 columns] [136 rows x 2 columns]
%% Cell type:code id:2da93d63 tags: %% Cell type:code id:2da93d63 tags:
``` python ``` python
# create figure and axis objects with subplots() # create figure and axis objects with subplots()
fig,ax1 = plt.subplots() fig,ax1 = plt.subplots()
ax2 = ax1.twinx() ax2 = ax1.twinx()
# make a plot # make a plot
ax1.plot(km_avg, Messwert_avg, color="red") # this line won't show up when there is a limit on the x axis ax1.plot(km_avg, Messwert_avg, color="red") # this line won't show up when there is a limit on the x axis
ax2.plot(Stromkilometer2, Depth, color="blue") ax2.plot(Stromkilometer2, Depth, color="blue")
# x-axis # x-axis
ax1.set_xlabel("Stromkilometer", fontsize = 14) ax1.set_xlabel("Stromkilometer", fontsize = 14)
ax2.set_xlim(586,830) ax2.set_xlim(586,830)
plt.xticks(np.arange(550, 850, step=50)) plt.xticks(np.arange(550, 850, step=50))
plt.gca().invert_xaxis() plt.gca().invert_xaxis()
# y-axis labels # y-axis labels
ax1.set_ylabel("Messwert", color="red", fontsize=14) ax1.set_ylabel("Messwert", color="red", fontsize=14)
ax2.set_ylabel("Depth",color="blue",fontsize=14) ax2.set_ylabel("Depth",color="blue",fontsize=14)
ax2.invert_yaxis() ax2.invert_yaxis()
``` ```
%% Output %% Output
%% Cell type:code id:81a50a6c tags: %% Cell type:code id:81a50a6c tags:
``` python ``` python
``` ```
......
%% Cell type:code id:0973acfd tags:
``` python
import glob
import os
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
```
%% Cell type:markdown id:cb524102 tags:
## pre-processing ems data
%% Cell type:code id:d50e6e4e tags:
``` python
#os.getcwd()
os.chdir("C:\\Users\\Hannah Russell\\north_sea_estuaries_visualisations")
cwd = os.path.abspath(os.curdir)
ems_EFW_df_1 = pd.read_csv(cwd + '\data\input\ems\df_1\ems_EFW.csv') #includes SSC and ignition loss
ems_EMD_df_1 = pd.read_csv(cwd + '\data\input\ems\df_1\ems_EMD.csv') #includes SSC
```
%% Cell type:code id:dbdc2f3a tags:
``` python
ems_EFW_df_1.head()
```
%% Output
sample# CET UTM East UTM North \
0 E001 28-8-2018 07:19 372825 5910331
1 E002 28-8-2018 07:23 372826 5910330
2 E003 28-8-2018 07:31 372826 5910329
3 E004 28-8-2018 07:34 372829 5910330
4 E005 28-8-2018 07:45 372829 5910330
.. ... ... ... ...
200 SB_EFW_097 24-01-2019 21:17:25 372898 5910337
201 SB_EFW_098 24-01-2019 21:32:19 372902 5910332
202 SB_EFW_099 24-01-2019 21:33:21 372904 5910331
203 SB_EFW_100 24-01-2019 21:46:59 372949 5910340
204 SB_EFW_101 24-01-2019 21:48:00 372956 5910343
waterdepth [m] SSC [mg/l] ignition loss [%] vert. Profile
0 298 1202 143 1.0
1 590 3183 145 NaN
2 297 1603 166 2.0
3 595 2843 139 NaN
4 302 921 186 3.0
.. ... ... ... ...
200 554 5447 123 NaN
201 296 5044 127 37.0
202 579 3759 131 NaN
203 297 4791 123 38.0
204 644 6399 121 NaN
[205 rows x 8 columns]
%% Cell type:code id:fa8847a7 tags:
``` python
ems_EMD_df_1.head()
```
%% Output
sample# CET UTM East (avg) UTM North (avg) \
0 1 28-08-2018 07:26:58 376996 5910984
1 2 28-08-2018 07:29:00 376996 5910984
2 3 28-08-2018 07:39:59 376996 5910984
3 4 28-08-2018 07:42:00 376996 5910984
4 5 28-08-2018 07:55:01 376996 5910984
waterdepth [m] SSC [mg/l]
0 60 320
1 25 230
2 64 300
3 25 250
4 60 240
%% Cell type:code id:c9ee6be7 tags:
``` python
```
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment