# Scraping the Web

NYC WTC documents portal

## wget: terminal HTTP request utility

```sh
wget -i gen-urls.txt --wait=2 --random-wait
```

## du and sort: disk usage info

the `-h` flag makes sure to use the human readable sorting rather than alphanumeric digit sorting

```sh
du -s * | sort -hr | awk '{print $1}' > size_distribution.csv
```

## pdfinfo: pages in docs

```sh
pdfinfo NYC-WTC_000000010.pdf | grep Pages | awk '{print $1}'
```

## bash scripting

We want to collect both filename and size, followed by sort

```sh
for file in ./nyc-wtc-docs/*; do
    # Check if it's a regular file (not a folder)
    if [ -f "$file" ]; then
        #echo "Processing: $file"
        pageCount=$(pdfinfo "$file" | grep "Pages" | awk '{print $2}')
        echo "$pageCount    $file" >> page_distribution_names.csv
    fi
done
```

## pdftoppm: operations on pdfs

```sh
pdftoppm -png NYC-WTC_000115336.pdf NYC-WTC_000115336
```

## ollama analysis

```sh
ollama run ornith-1.5:9b

./urban-data/static/public/NYC-WTC_000115336-1.png
```