Skip to content
Snippets Groups Projects
Commit a8cfe624 authored by Gjorgji's avatar Gjorgji
Browse files

traffic_simulation_Mar_20

parents
No related branches found
No related tags found
No related merge requests found
Showing
with 1485 additions and 0 deletions
# Default ignored files
/shelf/
/workspace.xml
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="CompilerConfiguration">
<annotationProcessing>
<profile name="Maven default annotation processors profile" enabled="true">
<sourceOutputDir name="target/generated-sources/annotations" />
<sourceTestOutputDir name="target/generated-test-sources/test-annotations" />
<outputRelativeToContentRoot value="true" />
<module name="simserver" />
</profile>
</annotationProcessing>
</component>
<component name="JavacSettings">
<option name="ADDITIONAL_OPTIONS_OVERRIDE">
<module name="simserver" options="-parameters" />
</option>
</component>
</project>
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="Encoding">
<file url="file://$PROJECT_DIR$/src/main/java" charset="UTF-8" />
</component>
</project>
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="RemoteRepositoriesConfiguration">
<remote-repository>
<option name="id" value="matsim" />
<option name="name" value="MATSim release repository" />
<option name="url" value="https://repo.matsim.org/repository/matsim" />
</remote-repository>
<remote-repository>
<option name="id" value="central" />
<option name="name" value="Central Repository" />
<option name="url" value="https://repo.maven.apache.org/maven2" />
</remote-repository>
<remote-repository>
<option name="id" value="central" />
<option name="name" value="Maven Central repository" />
<option name="url" value="https://repo1.maven.org/maven2" />
</remote-repository>
<remote-repository>
<option name="id" value="jboss.community" />
<option name="name" value="JBoss Community repository" />
<option name="url" value="https://repository.jboss.org/nexus/content/repositories/public/" />
</remote-repository>
</component>
</project>
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="ExternalStorageConfigurationManager" enabled="true" />
<component name="MavenProjectsManager">
<option name="originalFiles">
<list>
<option value="$PROJECT_DIR$/pom.xml" />
</list>
</option>
</component>
<component name="ProjectRootManager" version="2" languageLevel="JDK_18" default="true" project-jdk-name="openjdk-18" project-jdk-type="JavaSDK" />
</project>
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project version="4">
<component name="VcsDirectoryMappings">
<mapping directory="$PROJECT_DIR$" vcs="Git" />
</component>
</project>
\ No newline at end of file
FROM maven:3.8.1-openjdk-11-slim
WORKDIR /home/build
COPY pom.xml .
RUN mvn dependency:go-offline
RUN apt-get update -y && apt-get install -y r-base python3 python3-pip
RUN R -e "install.packages('ipfp', dependencies=TRUE)"
COPY .env .env
COPY src src/
RUN mvn clean package -Dmaven.test.skip=true -DskipTests
WORKDIR /home/simserver
RUN mv /home/build/target/*.jar ./app.jar
COPY requirements.txt ./
COPY population_generation ./population_generation
COPY .env .env
RUN pip3 install -r requirements.txt
EXPOSE 8082
EXPOSE 5432
ENTRYPOINT ["java","-jar","app.jar"]
README.md 0 → 100644
# URBANITE Traffic Simulation Component
This component creates and runs the traffic simulations.
## Database service connection
Traffic simulation component connects to the database of the Storage component.
Relevant environment variables:
- `database_url`
URL to storage component's database, including port and database name
- `databse_username`
Storage component's database username
- `database_password`
Storage component's database password
## Common data folder
URBANITE Components TS, DSS, and Storage share a common data folder.
The path to the data folder's initial contents is defined in environment variable `storage.data_dir`.
### `storage.data_dir` structure
Before running, populate the directory with data, available here:
[https://portal.ijs.si/nextcloud/s/m8zkQGAwGipS329](Storage component data)
This server has access to the URBANITE Storage component's shared data folder.
The structure is as follows:
- `data`
- `<city_id>`
- `before_simulation`
- population generation input files
- travel demand input files
- preprocessing files (for calibration)
- `original_input` - predefined scenarios and all their inputs
- `networks`
- `<net_id>` - predefined and user generated network go here
- `network.xml`
- `plans` - predefined and user generated plans go here since they are dependent on the network
- `<plan_id>`
- `plans.xml`
- `vehicles.xml` - predefined
- `config.xml` - original config that is copied to simulation folders
- `simulations`
- `<simulation_id>` - created when user creates simulation
- `network.xml`
- `vehicles.xml`
- `plans.xml`
- `config.xml`
- `results`
- `<datetime>` - create when user runs the simulation
## API
- GET `/city`
Returns list of all cities.
- GET `/city/selected`
Returns the city deployed inReturns the city deployed in
- GET `/city/<city_id>`
`city_id` is lowercase name of the city.
Returns the city entity data.
- GET `/city/<city_id>/scenarios`
Returns list of scenarios for city. **deprecated**
- GET `/city/<city_id>/coordinates`
Returns the city centre coordinates.
- GET `/city/<city_id>/networks`
Returns list of networks for city. **deprecated**
---
- GET `/network`
Returns list of all networks
- GET `/network/<net_id>`
Returns network entity object.
- POST `/network`
Creates a new network.
_payload_: NetworkDTO
Returns: NetworkDTO, including assigned ID
- DELETE `/network/<net_id>`
Deletes network.
- GET `/network/geojson/<net_id>`
Returns geojson representation of the network for visualization. **WIP**
---
- GET `/scenario`
Returns list of all scenarios.
- GET `/scenario/<scenario_id>`
Returns a scenario entity object.
- GET `/scenario/<scenario_id>/simulations`
Returns list of simulations in scenario. **DEPRECATED**
- POST `/scenario`
Create a scenario.
_Payload_: Scenario entity object
---
- GET `/simulation`
Returns all simulation entity objects
- GET `/simulation/<sim_id>/run`
Start the pipeline by simulation id.
1. preprocess population data
2. run population generation
3. run travel demand generation **ERROR**
4. run simulations
- GET `/simulation/<sim_id>`
Return simulation entity object by id
- GET `/simulation/<sim_id>/<file_name>`
Returns the file by name for simulation by id **DEPRECATED**
- POST `/simulation`
Create new simulation
_Payload_: SimulationDTO object JSON
Returns: SimulationDTO with assigned ID
- GET `/simulation/<sim_id>/data/<file_name>`
Returns the file by name for simulation by id **DEPRECATED**
- GET `/simulation/<sim_id>/data`
Returns the path of simulation's data folder **DEPRECATED**
## User actions
- page create simulation - **WIP**
1. create/choose network on UUI
2. create/choose plans for selected network on UUI
3. run simulation
- page results - **WIP**
1. list simulations
2. select simulation and visualize results
## Scripts and input/output files for before running simulation
- Bilbao:
- `bilbao_preprocess.py`
- Input:
- ES_2013h_EUSILC.csv, ES_2013p_EUSILC.csv, ES_2013d_EUSILC.csv, ES_2013r_EUSILC.csv
- numero_habitantes_distrito_barrio_edad_2020.csv
- ODM_bilbao1.csv
- Output:
- ind.csv
- cons.csv
- ODM_bilbao_districtLevel.csv
- ODM_bilbao_districtLevel_reduced.csv
- `bilbao_population.R`
- Input:
- ind.csv
- cons.csv
- Output:
- ind_full.csv
- `TravelDemandGenerationBilbao.java`
- Input:
- ODM_bilbao_districtLevel_codes.csv
- config.xml -> plans.xml, counts.xml, vehicle.xml, bilbao_multimodalnetwork.xml, schedule.xml
- ind_full.csv
- Output:
- plans.xml
__________________________________________________
- Amsterdam:
- `amsterdam_preprocess.py`
- Input:
- NL_2013h_EUSILC.csv, NL_2013p_EUSILC.csv, NL_2013d_EUSILC.csv, NL_2013r_EUSILC.csv
- 2021-22-gebieden-1-02.csv
- Output:
- #commented microdata_NL.csv
- ind.csv
- cons.csv
- `amsterdam_population.R`
- Input:
- ind.csv
- cons.csv
- Output:
- ind_full.csv
- `TravelDemandAmsterdam`:
- Input:
- config -> vehicles.xml. plans_cyclists.xml, amsterdam_fine.xml.gz
- ind_full.csv
- Output:
- plans.xml
__________________________________________________
- Helsinki:
- `helsinki_preprocess.py`
- Input:
- FI_2013h_EUSILC.csv, FI_2013p_EUSILC.csv, FI_2013d_EUSILC.csv, FI_2013r_EUSILC.csv
- Hki_vaesto_taulu9a.csv
- Output:
- microdata_NL.csv
- ind.csv
- cons.csv
- ferries.csv
- ferriesJatkasaari.csv
- `helsinki_population.R`
- Input:
- ind.csv
- cons.csv
- Output:
- indHelsinki.csv
- `TravelDemandGenerationHelsinki`
- Input:
- indHelsinki.csv
- config.xml -> helsinki_multimodalnetwork.xml, plans.xml, vehicles.xml, helsinki_schedule.xml
- ferriesJatkasaari.csv
- Output:
- plans.xml
This diff is collapsed.
services:
ts:
container_name: ts
build: .
ports:
- "${ts_port}:8082"
volumes:
- type: bind
source: ${host_data_dir}
target: ${data_dir}
networks:
- urbanite
networks:
urbanite:
external: true
name: urbanite
pom.xml 0 → 100644
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.4</version>
</parent>
<groupId>si.ijs.urbanite</groupId>
<artifactId>simserver</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>simserver</name>
<description>Server exposes the traffic simulation API.</description>
<properties>
<java.version>1.8</java.version>
<log4j2.version>2.15.0</log4j2.version>
</properties>
<repositories>
<repository>
<id>matsim</id>
<name>MATSim release repository</name>
<url>https://repo.matsim.org/repository/matsim</url>
</repository>
</repositories>
<dependencies>
<!-- load .env file into dotenv object -->
<dependency>
<groupId>me.paulschwarz</groupId>
<artifactId>spring-dotenv</artifactId>
<version>2.5.4</version>
</dependency>
<!-- Spring dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- Osmosis -->
<dependency>
<groupId>org.openstreetmap.osmosis</groupId>
<artifactId>osmosis-core</artifactId>
<version>0.48.3</version>
</dependency>
<dependency>
<groupId>org.openstreetmap.osmosis</groupId>
<artifactId>osmosis-xml</artifactId>
<version>0.48.3</version>
</dependency>
<dependency>
<groupId>org.openstreetmap.osmosis</groupId>
<artifactId>osmosis-pbf</artifactId>
<version>0.48.3</version>
</dependency>
<dependency>
<groupId>org.openstreetmap.osmosis</groupId>
<artifactId>osmosis-areafilter</artifactId>
<version>0.48.3</version>
</dependency>
<!-- Matsim -->
<dependency>
<groupId>org.matsim</groupId>
<artifactId>matsim</artifactId>
<version>12.0</version>
</dependency>
<dependency>
<groupId>org.matsim</groupId>
<artifactId>pt2matsim</artifactId>
<version>21.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.matsim.contrib/cadytsIntegration -->
<dependency>
<groupId>org.matsim.contrib</groupId>
<artifactId>cadytsIntegration</artifactId>
<version>12.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.matsim.contrib/emissions -->
<dependency>
<groupId>org.matsim.contrib</groupId>
<artifactId>emissions</artifactId>
<version>12.0</version>
</dependency>
<!-- data wrangling etc -->
<dependency>
<groupId>tech.tablesaw</groupId>
<artifactId>tablesaw-core</artifactId>
<version>0.38.3</version>
</dependency>
<dependency>
<groupId>tech.tablesaw</groupId>
<artifactId>tablesaw-json</artifactId>
<version>0.38.3</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20090211</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
<artifactId>json-simple</artifactId>
<version>1.1.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.orbisgis/poly2tri-core -->
<dependency>
<groupId>org.orbisgis</groupId>
<artifactId>poly2tri-core</artifactId>
<version>0.1.2</version>
</dependency>
<dependency>
<groupId>org.matsim</groupId>
<artifactId>pt2matsim</artifactId>
<version>21.10</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.4.1</version>
<configuration>
<mainClass>si.ijs.urbanite.simserver.SimserverApplication</mainClass>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
set.seed(10)
#this data was pre-processed using python script
data_path <- file.path("data", "amsterdam", "before_simulation")
ind <- read.csv(file.path(data_path, "ind.csv"))
cons <- read.csv(file.path(data_path, "cons.csv"))
colnames(cons) <- c('district',c(1:8),c('totals'))
#remove district and totals
totals <- cons["totals"]
districts <- cons[1]
districts <- as.data.frame(districts[-24,])
cons <- cons[1:23,-c(1,dim(cons)[2])]
#ind <- subset(ind,ind$age>9)
ind$age <- as.character(ind$age)
#flattering individual data
cat_age <- model.matrix(~ ind$age-1)
#don't have constraints for gender
#cat_gender <- model.matrix(~ ind$gender - 1)[,c(2,1)]
ind_cat <- cbind(cat_age)
ind_agg <- colSums(ind_cat)
rbind(cons[1,], ind_agg)
#create initial weight matrix
weights <- matrix(data=1,nrow = nrow(ind), ncol = nrow(cons))
#remove age group 1 from cons since no individuals in the sample data are in that group
cons <- cons[,-1]
#apply ipf
library(ipfp)
cons <- apply(cons, 2, as.numeric)
weights_maxit_2 <- weights # create a copy of the weights object
for(i in 1:ncol(weights)){
weights_maxit_2[,i] <- ipfp(cons[i,], t(ind_cat), rep(1,nrow(ind)), maxit = 2)
}
n_dist <- nrow(districts)
data <-ind
data <- cbind(data,w=weights_maxit_2[,1])
data <- cbind(data,dist=1)
for (i in 2:n_dist){
p <- ind
p <- cbind(p,w=weights_maxit_2[,i])
p <- cbind(p,dist=i)
data <- rbind(data,p)
}
#integerisation of weights
int_trs <- function(x){
# For generalisation purpose, x becomes a vector
xv <- as.vector(x) # allows trs to work on matrices
xint <- floor(xv) # integer part of the weight
r <- xv - xint # decimal part of the weight
def <- round(sum(r)) # the deficit population
# the weights be 'topped up' (+ 1 applied)
topup <- sample(length(x), size = def, prob = r)
xint[topup] <- xint[topup] + 1
dim(xint) <- dim(x)
dimnames(xint) <- dimnames(x)
xint
}
int_weights <- int_trs(data$w)
data <- cbind(data,int_weights)
data$X <- c(1:514395)
#expansion
int_expand_vector <- function(x){
index <- 1:length(x)
rep(index, round(x))
}
exp_indices <- int_expand_vector(int_weights)
ind_full <- data[exp_indices,c( "age", "gender", "dist")]
ind_full <- data.frame(ind_full)
write.csv(ind_full, file.path(data_path, "ind_full.csv"), quote=FALSE)
#!/usr/bin/env python
# coding: utf-8
# Extract data from EUSILC microdata
#
# Since the demographic statistics are not available for every age, population is devided into age groups like the following:
# 1. 0-3
# 2. 4-12
# 3. 13-17
# 4. 18-22
# 5. 23-24
# 6. 25-49
# 7. 50-64
# 8. 65+
#
# For that porpuse the microdata age statiscis will be replaced with groups insted of actual ages.
# In[80]:
import pandas as pd
import numpy as np
from os.path import join as joinpath
data_path = joinpath("data", "amsterdam", "before_simulation")
household = "NL_2013h_EUSILC.csv"
personal = "NL_2013p_EUSILC.csv"
household_register = "NL_2013d_EUSILC.csv"
personal_register = "NL_2013r_EUSILC.csv"
#variables we need from files
d_var = ["DB030","DB090"]#hhid, federal state, hh weight
p_var = ["PB030","PL031"] #economic status
r_var = ["RB030","RB090","RB050","RB080"] #person's id, person's gender, weights, year of birth
#read data and merge
h = pd.read_csv(joinpath(data_path, household))
p = pd.read_csv(joinpath(data_path, personal))[p_var]
d = pd.read_csv(joinpath(data_path, household_register))[d_var]
r = pd.read_csv(joinpath(data_path, personal_register))[r_var]
r["hhid"] = (np.floor(r["RB030"]/100)).astype(int)
r["RB080"] = 2022-r["RB080"]
r = r.merge(d, left_on = "hhid", right_on = "DB030")
# r.to_csv(joinpath(data_path, "microdata_NL.csv"))
# add grpup og age instead age
age = r["RB080"]
for i in range(0,4):
age = age.replace(i,1)
for i in range(4,13):
age = age.replace(i,2)
for i in range(13,18):
age = age.replace(i,3)
for i in range(18,23):
age = age.replace(i,4)
for i in range(23,25):
age = age.replace(i,5)
for i in range(25,50):
age = age.replace(i,6)
for i in range(50,65):
age = age.replace(i,7)
for i in range(65,120):
age = age.replace(i,8)
r["RB080"] = age
ind = r[["RB090","RB080"]]
ind = ind.rename(columns={"RB090":"gender","RB080":"age"})
ind["gender"] = ind["gender"].replace({2:"f",1:"m"})
ind.to_csv(joinpath(data_path, "ind.csv"), sep=",")
# Filter marginal data to fit the IPF algorithm
# In[138]:
cons = pd.read_csv(joinpath(data_path, "2021-22-gebieden-1-02.csv"))
cons = cons.dropna()
cons=cons.rename(columns={"0-3":"1","12-Mar":"2","13-17":"3","18-22":"4","23-24":"5","25-49":"6","50-64":"7","65+":"8"})
# In[139]:
districts_lvl1 = ["A Centrum","E West","F Nieuw-West", "K Zuid","M Oost","N Noord","T Zuidoost"]
for d in districts_lvl1:
cons = cons[cons["gb/std"] !=d]
cons.columns = cons.columns.astype(str)
cons.to_csv(joinpath(data_path, "cons.csv"), sep=',',index=False)
set.seed(10)
#this data was pre-processed using python script
data_path <- file.path("data", "bilbao", "before_simulation")
ind <- read.csv(file.path(data_path, "ind.csv"))
cons <- read.csv(file.path(data_path, "cons.csv"))
colnames(cons) <- c('district',c(10:89),c('f','m','totals'))
#remove district and totals
totals <- cons["totals"]
districts <- cons[1]
cons <- cons[,-c(1,dim(cons)[2])]
ind <- subset(ind,ind$age>9)
ind$age <- as.character(ind$age)
#flattering individual data
cat_age <- model.matrix(~ ind$age-1)
cat_gender <- model.matrix(~ ind$gender - 1)[,c(2,1)]
ind_cat <- cbind(cat_age, cat_gender)
ind_agg <- colSums(ind_cat)
rbind(cons[1,], ind_agg)
#create initial weight matrix
weights <- matrix(data=1,nrow = nrow(ind), ncol = nrow(cons))
#apply ipf
library("ipfp")
cons <- apply(cons, 2, as.numeric)
weights_maxit_2 <- weights # create a copy of the weights object
for(i in 1:ncol(weights)){
weights_maxit_2[,i] <- ipfp(cons[i,], t(ind_cat), rep(1,nrow(ind)), maxit = 2)
}
n_dist <- nrow(districts)
data <-ind
data <- cbind(data,w=weights_maxit_2[,1])
data <- cbind(data,dist=1)
for (i in 2:n_dist){
p <- ind
p <- cbind(p,w=weights_maxit_2[,i])
p <- cbind(p,dist=i)
data <- rbind(data,p)
}
#integerisation of weights
int_trs <- function(x){
# For generalisation purpose, x becomes a vector
xv <- as.vector(x) # allows trs to work on matrices
xint <- floor(xv) # integer part of the weight
r <- xv - xint # decimal part of the weight
def <- round(sum(r)) # the deficit population
# the weights be 'topped up' (+ 1 applied)
topup <- sample(length(x), size = def, prob = r)
xint[topup] <- xint[topup] + 1
dim(xint) <- dim(x)
dimnames(xint) <- dimnames(x)
xint
}
int_weights <- int_trs(data$w)
data <- cbind(data,int_weights)
data$X <- c(1:dim(data)[1])
#expansion
int_expand_vector <- function(x){
index <- 1:length(x)
rep(index, round(x))
}
exp_indices <- int_expand_vector(int_weights)
ind_full <- data[exp_indices,c( "age", "gender", "dist")]
ind_full <- data.frame(ind_full)
write.csv(ind_full, file.path(data_path, "ind_full.csv"), quote=FALSE)
#!/usr/bin/env python
# coding: utf-8
# ### EUROSTAT data - sample data
#
# Description of EUROSTAT variables values:
#
# 1. (RB090)gender: 1- male, 2-female
# 2. (RB080)age
# 3. (PD060) Regularly participate in a leisure activity:
# 1 Yes
# No - cannot afford it
# No - other reason
# 4. (PL031) Self-defined current economic status:
# 1 Employee working full-time
# 2 Employee working part-time
# 3 Self-employed working full-time (including family worker)
# 4 Self-employed working part-time (including family worker)
# 5 Unemployed
# 6 Pupil, student, further training, unpaid work experience
# 7 In retirement or in early retirement or has given up business
# 8 Permanently disabled or/and unfit to work
# 9 In compulsory military or community service
# 10 Fulfilling domestic tasks and care responsibilities
# 11 Other inactive person
# 5. (PE040) Highest ISCED level attained:
# 0 pre-primary education
# 1 primary education
# 2 lower secondary education
# 3 (upper) secondary education
# 4 post-secondary non tertiary education
# 5 first stage of tertiary education (not leading directly to an
# advanced research qualification)
# 6 second stage of tertiary
# In[ ]:
import pandas as pd
import numpy as np
from os.path import join as joinpath
data_path = joinpath("data", "bilbao", "before_simulation")
sample = joinpath(data_path)
household = "ES_2013h_EUSILC.csv"
personal = "ES_2013p_EUSILC.csv"
household_register = "ES_2013d_EUSILC.csv"
personal_register = "ES_2013r_EUSILC.csv"
# variables we need from files
d_var = ["DB030", "DB040", "DB090"] # hhid, federal state, hh weight
# person's id, REGULARLY PARTICIPATE IN A LEISURE ACTIVITY, SELF-DEFINED CURRENT ECONOMIC STATUS,HIGHEST ISCED LEVEL ATTAINED
p_var = ["PB030", "PD060", "PL031", "PE040"]
# person's id, person's gender, weights, year of birth
r_var = ["RB030", "RB090", "RB050", "RB080"]
# read data and merge
h = pd.read_csv(joinpath(sample, household))
p = pd.read_csv(joinpath(sample, personal))[p_var]
d = pd.read_csv(joinpath(sample, household_register))[d_var]
r = pd.read_csv(joinpath(sample, personal_register))[r_var]
r["hhid"] = (np.floor(r["RB030"]/10)).astype(int)
r["RB080"] = 2021-r["RB080"]
r = r.merge(d, left_on="hhid", right_on="DB030")
#r = r.merge(p,left_on = "RB030", right_on="PB030")
ind = r[["RB090", "RB080"]]
ind = ind.rename(columns={"RB090": "gender", "RB080": "age"})
ind["gender"] = ind["gender"].replace({2: "f", 1: "m"})
ind.to_csv(joinpath(data_path, "ind.csv"), sep=",")
# ### Aggregated data
#
# Preprocess it to fit the IPF algorithm
# In[ ]:
cons = pd.read_csv(joinpath(data_path, "numero_habitantes_distrito_barrio_edad_2020.csv"), sep=';')
cons = cons.drop(cons.columns[[0,3,4]],1)
# save totals column
cons = cons[1:46].groupby(cons.columns[1])[cons.columns[2:]].sum()
totals = cons["TOTAL BARRIO"]
cons = cons.drop("TOTAL BARRIO", axis=1)
cons = cons.drop(cons.columns[0:20], axis=1)
cons = cons.drop(cons.columns[160:224], axis=1)
males = cons[cons.columns[::2]]
females = cons[cons.columns[1::2]]
females = females.sum(axis=1)
males = males.sum(axis=1)
cons = cons.groupby((np.arange(len(cons.columns)) // 2) + 1, axis=1).sum()
cons['f'] = females
cons['m'] = males
cons['totals'] = totals
cons.columns = cons.columns.astype(str)
# save
sum_cons = cons.iloc[0:2, ].sum(axis=0)
sum_cons = sum_cons.rename("ABANDO", axis=1)
new_cons = cons.iloc[2:, ]
new_cons = new_cons.append(sum_cons)
cons = new_cons
cons.to_csv(joinpath(data_path, "cons.csv"), sep=',')
# ### Filter amenities
# In[ ]:
# amenities:
sustence = ["bar", "cafe", "biergarten", "cafe", "fast_food",
"food_court", "ice_cream", "pub", "restaurant"]
education = ["college", "driving_school", "kindergarten", "language_school",
"library", "toy_library", "music_school", "school", "university"]
financial = ["bank"]
healthcare = ["clinic", "dentist", "doctors", "hospital",
"nursing_home", "pharmacy", "social_facility", "veterinary"]
entertainment = ["arts_center", "casino", "cinema", "community_centre", "conference_centre",
"events_venue", "nightclub", "planetarium", "social_centre", "studio", "theatre"]
public_services = ["courthouse", "embassy", "fire_station", "police", "post_box",
"post_depot", "post_office", "prison", "ranger_station", "townhall"]
others = ["marketplace"]
sustence = {
"anemity": "",
"lat": "",
"long": ""
}
# read the osm file
osm = open(joinpath(data_path, "amenities_bilbao.osm"))
# TODO: extract location from
# ### Preprocess OD matrices
#
# - districts to be used: ![image.png](attachment:image.png)
# In[ ]:
odm = pd.read_csv(joinpath(data_path, "ODM_bilbao1.csv"))
odm['From location'].unique()
odm['To location'].unique()
# districts
DEUSTO = ['D1 NORTE', 'D1 SUR']
URIBARRI = ["CASTANOS", "D2 ALTO", "RIA ESTE",
"RIA OESTE", "MB L3 (METRO BILBAO LINEA 3)"]
OTXARKOAGA_TXURDINAGA = ["OTXARKOAGA", "TXURDINAGA"]
BEGONA = ["D4 SUR"]
IBAIONDO = ["CASCO VIEJO", "D5 ESTE", "D5 OESTE"]
ABANDO = ['CENTRO', 'HOSPITAL', 'ACCESOS V (ACCESOS VIARIOS)', 'INTERMODAL']
ERREKALDE = ["D7 NORTE", "D7 SUR"]
BASURT_ZORROTZA = ["BASURTO", "ZORROZA"]
districts = ["ABANDO",
"BASURT_ZORROTZA",
"BEGONA",
"DEUSTO",
"ERREKALDE",
"IBAIONDO",
"OTXARKOAGA_TXURDINAGA",
"URIBARRI"]
for i in DEUSTO:
odm = odm.replace(to_replace=str(i), value="DEUSTO")
for i in URIBARRI:
odm = odm.replace(to_replace=str(i), value="URIBARRI")
for i in OTXARKOAGA_TXURDINAGA:
odm = odm.replace(to_replace=str(i), value="OTXARKOAGA_TXURDINAGA")
for i in BEGONA:
odm = odm.replace(to_replace=str(i), value="BEGONA")
for i in IBAIONDO:
odm = odm.replace(to_replace=str(i), value="IBAIONDO")
for i in ABANDO:
odm = odm.replace(to_replace=str(i), value="ABANDO")
for i in ERREKALDE:
odm = odm.replace(to_replace=str(i), value="ERREKALDE")
for i in BASURT_ZORROTZA:
odm = odm.replace(to_replace=str(i), value="BASURT_ZORROTZA")
odm.to_csv(joinpath(data_path, "ODM_bilbao_districtLevel.csv"))
# Merge rows by column name to reduce num of rows
odm = odm.groupby(['From location', 'To location']).sum()
odm.to_csv(joinpath(data_path, "ODM_bilbao_districtLevel_reduced.csv"))
set.seed(10)
#this data was pre-processed using python script
data_path <- file.path("data", "helsinki", "before_simulation")
ind <- read.csv(file.path(data_path, "ind.csv"))
cons <- read.csv(file.path(data_path, "cons.csv"))
colnames(cons) <- c('district',c(1:7),c('m','f','totals'))
#remove district
districts <- cons[1]
cons <- cons[,-c(1)]
ind$age <- as.character(ind$age)
#flattering individual data
cat_age <- model.matrix(~ ind$age-1)
cat_gender <- model.matrix(~ ind$gender - 1)[,c(2,1)]
ind_cat <- cbind(cat_age, cat_gender)
ind_agg <- colSums(ind_cat)
#create initial weight matrix
weights <- matrix(data=1,nrow = nrow(ind), ncol = nrow(cons))
#apply ipf
library(ipfp)
cons <- apply(cons, 2, as.numeric)
weights_maxit_2 <- weights # create a copy of the weights object
for(i in 1:ncol(weights)){
weights_maxit_2[,i] <- ipfp(cons[i,], t(ind_cat), rep(1,nrow(ind)), maxit = 2)
}
n_dist <- nrow(districts)
data <-ind
data <- cbind(data,w=weights_maxit_2[,1])
data <- cbind(data,dist=1)
for (i in 2:n_dist){
p <- ind
p <- cbind(p,w=weights_maxit_2[,i])
p <- cbind(p,dist=i)
data <- rbind(data,p)
}
#integerisation of weights
int_trs <- function(x){
# For generalisation purpose, x becomes a vector
xv <- as.vector(x) # allows trs to work on matrices
xint <- floor(xv) # integer part of the weight
r <- xv - xint # decimal part of the weight
def <- round(sum(r)) # the deficit population
# the weights be 'topped up' (+ 1 applied)
topup <- sample(length(x), size = def, prob = r)
xint[topup] <- xint[topup] + 1
dim(xint) <- dim(x)
dimnames(xint) <- dimnames(x)
xint
}
int_weights <- int_trs(data$w)
data <- cbind(data,int_weights)
data$X <- c(1:187664)
#expansion
int_expand_vector <- function(x){
index <- 1:length(x)
rep(index, round(x))
}
exp_indices <- int_expand_vector(int_weights)
ind_full <- data[exp_indices,c( "age", "gender", "dist")]
ind_full <- data.frame(ind_full)
write.csv(ind_full, file.path(data_path,"indHelsinki.csv"), quote=FALSE)
#!/usr/bin/env python
# coding: utf-8
# ### Sample data
# Age groups:
# 1. 0-18
# 2. 18-25
# 3. 25-35
# 4. 35-45
# 5. 45-55
# 6. 55-65
# 7. 65+
#
#
# In[2]:
import pandas as pd
import numpy as np
import os
from os.path import join as joinpath
data_path = joinpath("data", "helsinki", "before_simulation")
sample = joinpath(data_path)
household = "FI_2013h_EUSILC.csv"
personal = "FI_2013p_EUSILC.csv"
household_register = "FI_2013d_EUSILC.csv"
personal_register = "FI_2013r_EUSILC.csv"
#variables we need from files
d_var = ["DB030","DB090"]#hhid, federal state, hh weight
p_var = ["PB030","PL031"] #economic status
r_var = ["RB030","RB090","RB050","RB080"] #person's id, person's gender, weights, year of birth
#read data and merge
h = pd.read_csv(joinpath(sample, household))
p = pd.read_csv(joinpath(sample, personal))[p_var]
d = pd.read_csv(joinpath(sample, household_register))[d_var]
r = pd.read_csv(joinpath(sample, personal_register))[r_var]
r["hhid"] = (np.floor(r["RB030"]/100)).astype(int)
r["RB080"] = 2022-r["RB080"]
r = r.merge(d, left_on = "hhid", right_on = "DB030")
r.to_csv(joinpath(data_path, "microdata_NL.csv"))
# add grpup og age instead age
age = r["RB080"]
for i in range(0,18):
age = age.replace(i,1)
for i in range(18,25):
age = age.replace(i,2)
for i in range(25,35):
age = age.replace(i,3)
for i in range(35,45):
age = age.replace(i,4)
for i in range(45,55):
age = age.replace(i,5)
for i in range(55,65):
age = age.replace(i,6)
for i in range(65,120):
age = age.replace(i,7)
r["RB080"] = age
ind = r[["RB090","RB080"]]
ind = ind.rename(columns={"RB090":"gender","RB080":"age"})
ind["gender"] = ind["gender"].replace({2:"f",1:"m"})
ind.to_csv(os.sep.join([data_path, "ind.csv"]),sep=",")
# ### Marginal data
# In[117]:
import pandas as pd
import numpy as np
cons = pd.read_csv(os.sep.join([data_path, "Hki_vaesto_taulu9a.csv"]))
# gender = cons[3,]
districts = ["1 Eteläinen","2 Läntinen","3 Keskinen","4 Pohjoinen","5 Koillinen","6 Kaakkoinen","7 Itäinen","8 Östersundom"]
# pd.to_numeric(cons)
# make age groups by merging columns:
# cons[2:].groupby("Ikä, vuotta").sum()
a = cons[2:]
a = a.apply(pd.to_numeric)
a = a.groupby("Ikä, vuotta").sum()
aa = a.groupby((np.arange(len(a.columns)) // 2) + 1, axis=1).sum()
aa = aa.T
aa.reset_index(inplace=True)
aa.drop("index",axis=1,inplace=True)
gender = cons[1:2].drop("Ikä, vuotta",axis=1)
gender = gender.T
gender.reset_index(inplace=True)
gender.drop("index",axis=1,inplace=True)
df = pd.DataFrame(gender.values.reshape(-1, 2), columns = ['males', 'females'])
cons = pd.concat([aa,df], axis=1,join = 'outer')
cons.index = districts
cons.males = cons.males.astype(int)
cons.females = cons.females.astype(int)
cons['total'] = cons.males + cons.females
cons.to_csv(os.sep.join([data_path, "cons.csv"]),sep=",")
# ### Ferry arrivals data
# In[133]:
import os
import json
import pandas as pd
ferries_path = os.sep.join([data_path, "ferries.json"])
ferries = open(ferries_path)
data = json.load(ferries)
row = []
for i in range(0,len(data)):
hour = data[i]['startDate'][12:13]
minutes = data[i]['startDate'][14:16]
time = int(hour)*60*60 + int(minutes)*60
row.append([data[i]['terminal'],time,data[i]['passengerCount'],data[i]['departsFromHarbour'],data[i]['arrivesToHarbour'],data[i]['subCategory']])
ferries_csv = pd.DataFrame(row,columns=["terminal","time (sec)","passCount","departsFrom","arrivesTo","category"])
ferries_Jatkasaari = ferries_csv.loc[ferries_csv['terminal'] == 'West Terminal 2']
ferries_csv.to_csv(os.sep.join([data_path, 'ferries.csv']))
ferries_Jatkasaari.to_csv(os.sep.join([data_path, 'ferriesJatkasaari.csv']))
set.seed(10)
#this data was pre-processed using python script
data_path <- file.path("data","messina", "before_simulation")
ind <- read.csv(file.path(data_path, "ind.csv"))
cons <- read.csv(file.path(data_path, "cons.csv"))
colnames(cons) <- c('district',c(1:5),c('f','m','totals'))
#remove district and totals
districts <- cons[1]
#cons <- cons[,-c(1)]
ind$age <- as.character(ind$age)
#flattering individual data
cat_age <- model.matrix(~ ind$age-1)
cat_gender <- model.matrix(~ ind$gender - 1)[,c(2,1)]
ind_cat <- cbind(cat_age, cat_gender)
ind_agg <- colSums(ind_cat)
rbind(cons[1,], ind_agg)
#create initial weight matrix
weights <- matrix(data=1,nrow = nrow(ind), ncol = nrow(cons))
#apply ipf
library(ipfp)
cons <- apply(cons, 2, as.numeric)
weights_maxit_2 <- weights # create a copy of the weights object
for(i in 1:ncol(weights)){
weights_maxit_2[,i] <- ipfp(cons[i,], t(ind_cat), rep(1,nrow(ind)), maxit = 2)
}
n_dist <- nrow(districts)
data <-ind
data <- cbind(data,w=weights_maxit_2[,1])
data <- cbind(data,dist=1)
for (i in 2:n_dist){
p <- ind
p <- cbind(p,w=weights_maxit_2[,i])
p <- cbind(p,dist=i)
data <- rbind(data,p)
}
#integerisation of weights
int_trs <- function(x){
# For generalisation purpose, x becomes a vector
xv <- as.vector(x) # allows trs to work on matrices
xint <- floor(xv) # integer part of the weight
r <- xv - xint # decimal part of the weight
def <- round(sum(r)) # the deficit population
# the weights be 'topped up' (+ 1 applied)
topup <- sample(length(x), size = def, prob = r)
xint[topup] <- xint[topup] + 1
dim(xint) <- dim(x)
dimnames(xint) <- dimnames(x)
xint
}
int_weights <- int_trs(data$w)
data <- cbind(data,int_weights)
data$X <- c(1:dim(data)[1])
#expansion
int_expand_vector <- function(x){
index <- 1:length(x)
rep(index, round(x))
}
exp_indices <- int_expand_vector(int_weights)
ind_full <- data[exp_indices,c( "age", "gender", "dist")]
ind_full <- data.frame(ind_full)
write.csv(ind_full, file.path(data_path,"ind_full.csv"), quote=FALSE)
#!/usr/bin/env python
# coding: utf-8
# Age groups:
# 1. 0-19
# 2. 20-44
# 3. 45-64
# 4. 65+
# In[7]:
import pandas as pd
import numpy as np
import os
data_path = os.sep.join(["data","messina", "before_simulation"])
sample_data = os.sep.join([data_path])
household = "IT_2013h_EUSILC.csv"
personal = "IT_2013p_EUSILC.csv"
household_register = "IT_2013d_EUSILC.csv"
personal_register = "IT_2013r_EUSILC.csv"
#variables we need from files
d_var = ["DB030","DB040","DB090"]#hhid, federal state, hh weight
p_var = ["PB030","PL031"] #economic status
r_var = ["RB030","RB090","RB050","RB080"] #person's id, person's gender, weights, year of birth
#read data and merge
h = pd.read_csv(os.sep.join([sample_data,household]))
p = pd.read_csv(os.sep.join([sample_data,personal]))[p_var]
d = pd.read_csv(os.sep.join([sample_data,household_register]))[d_var]
r = pd.read_csv(os.sep.join([sample_data,personal_register]))[r_var]
r["hhid"] = (np.floor(r["RB030"]/10)).astype(int)
r["RB080"] = 2021-r["RB080"]
r = r.merge(d, left_on = "hhid", right_on = "DB030")
# r.to_csv(data_path+"microdata_ES.csv")
#
# add grpup og age instead age
age = r["RB080"]
for i in range(0,19):
age = age.replace(i,1)
for i in range(20,24):
age = age.replace(i,2)
for i in range(25,44):
age = age.replace(i,3)
for i in range(45,64):
age = age.replace(i,4)
for i in range(65,120):
age = age.replace(i,5)
r["RB080"] = age
ind = r[["RB090","RB080"]]
ind = ind.rename(columns={"RB090":"gender","RB080":"age"})
ind["gender"] = ind["gender"].replace({2:"f",1:"m"})
ind.to_csv(os.sep.join([data_path,"ind.csv"]),sep=",")
# Filter marginal data
# In[110]:
# Read json marginal data
import json
marginal_path = os.sep.join([data_path,'marginal_data.json'])
f = open(marginal_path)
marginal_data = json.load(f)
header = list()
for i in marginal_data["districts-summary"]["district-I"]["ages"]:
header.append(i)
header.append("m")
header.append("f")
data = pd.DataFrame(columns=header)
for i in marginal_data["districts-summary"]:
row=list()
males = 0
females = 0
for j in marginal_data["districts-summary"][i]["ages"]:
row.append(marginal_data["districts-summary"][i]["ages"][j]["M"]+marginal_data["districts-summary"][i]["ages"][j]["F"])
males+=marginal_data["districts-summary"][i]["ages"][j]["F"]
females+=marginal_data["districts-summary"][i]["ages"][j]["M"]
row.append(males)
row.append(females)
a_series = pd.Series(row, index = data.columns)
data = data.append(a_series,ignore_index=True)
# Group age to age groups
# In[122]:
cons=data
a1 = cons['0']+cons['1-4']+cons['5-9']+cons['10-14']+cons['15-19']
a2 = cons['20-24']
a3 = cons['25-29']+cons['30-34']+cons['35-39']+cons['40-44']
a4 = cons['45-49']+cons['50-54']+cons['55-59']+cons['60-64']
a5 = cons['70-74']+cons['75-79']+cons['80-84']+cons['85-89']+cons['90-94']+cons['95-99']+cons['>100']
cons1 = pd.DataFrame()
# cons1['district'] = cons['district']
cons1['1'] = a1
cons1['2'] = a2
cons1['3'] = a3
cons1['4'] = a4
cons1['5'] = a5
cons1['f'] = cons['f']
cons1['m'] = cons['m']
cons1['total']=cons1.m+cons1.f
cons1.to_csv(os.sep.join([data_path,"cons.csv"]))
Population, travel demand generation and calibration (Bilbao case)
Link for data: https://onedrive.live.com/?id=A8EDE1F35936A490%2149811&cid=A8EDE1F35936A490
1. Population generation
1. preprocessing python (preprocess.ipynb)
Input data:
1. Sample data: ES_2013_EUSILC
2. Aggregated data: numero_habitantes_distrito_barrio_edad_2020.csv
3. ODM_vilbao1.csv (used for travel demand)
Output data:
1. ind.csv - sample data from different tables containing variables of interest
2. cons.csv - preprocessed aggretagted data to fit the IPF algorithm
3. ODM_bilbao_districtLevel_reduced.csv
2. Synthetic population generation (bilbao_population.R)
Input data:
1. ind.csv
2. cons.csv
Output data:
1. ind_full.csv
2. Travel demand generation (java)
1. Generatin plans:
Input data:
1. ODM_bilbao_districtLevel_reduced.csv
2. config file (MATSIM)
3. network file
4. ind_full.csv
Output data
2. travel demand (10%)
3. Calibration
cadyts module is used to calibrate the model using traffic counts
To calibrate the simulation, cadyts integration tool is used. Required input file is counts.xml. The calibration is configured in the config file before running the simulation
5. Emissions
Emission module is configured for the following traffic situations:
1. linkSpeed >=80 km/h URB/MW-City/80/Freeflow
2. 60<=linkSpeed<80 URB/Trunk-City/60/Freeflow
3. 50<=linkSpeed<60 URB/Local/50/Freeflow
4. linkSpeed<50 URB/Access/30/Freeflow
========================
Different categories of vehicles:
1. pass.car ---> PASSENGER_CAR (petrol (4S),&gt;=2L,PC-P-Euro-1)
3. bus - use same weights as PASSNGEE_CAR since no weights can be obtained for cold start emissions for urban bus
The emissions are configured in the config file. All input files are in input_emissions folder.
The output represents events with amount of pollution on the links.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment