Error on training? json.decoder.JSONDecodeError


#1

I've downloaded games for ML bot training via hlt client, compressed it as zip and get following output, additionally I also tried without compressing and renaming to start with replay, same error with json.

~/halite/mlbot🍀$ python3 -m tsmlstarterbot.train --model_name model_long_training --data data/replays.zip --games_limit 100 --steps 5000 --seed 15485863
Found 230 games.
Trying to load up to 100 games ...
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 193, in _run_module_as_main
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
File "/Users/Dovla/Desktop/Dev/halite/mlbot/tsmlstarterbot/train.py", line 123, in <module>
main()
File "/Users/Dovla/Desktop/Dev/halite/mlbot/tsmlstarterbot/train.py", line 81, in main
raw_data = fetch_data_zip(args.data, args.games_limit)
File "/Users/Dovla/Desktop/Dev/halite/mlbot/tsmlstarterbot/train.py", line 56, in fetch_data_zip
d = json.loads(lines[0].decode())
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

What am I doing wrong?


#2

Did you try using the make commands in the makefile to train, I havent tested this by directly calling the python code yet.


#3

Yes, initially I train with make and it works. Then I download games I want, zip them, and change the name of in the Makefile, run make and I think its same error, line before last?

~/Desktop/Dev/halite/mlbot🍀 $ make
mkdir -p models/
python3 -m tsmlstarterbot.train --model_name model_long_training --data data/replays.zip --games_limit 1000 --steps 5000 --seed 15485863
Found 230 games.
Trying to load up to 1000 games ...
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/Dovla/Desktop/Dev/halite/mlbot/tsmlstarterbot/train.py", line 123, in <module>
main()
File "/Users/Dovla/Desktop/Dev/halite/mlbot/tsmlstarterbot/train.py", line 81, in main
raw_data = fetch_data_zip(args.data, args.games_limit)
File "/Users/Dovla/Desktop/Dev/halite/mlbot/tsmlstarterbot/train.py", line 56, in fetch_data_zip
d = json.loads(lines[0].decode())
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
make: *** [models/model_long_training.ckpt.meta] Error 1

#4

Anyone has an idea what might be the reason? Because make with default file works, must be something in way I download files?


#5

When you download new files, have you checked whether you have decoded them into json?


#6

hmmm.. no I have not done so. The only thing I did was download, and zip them.

Where can I read about this part? Somehow I missed it.
How is it usually done?

Thanks


#7

Anyone has a potential idea?


#8

You are right, it was due to json. The begging of the file has

b'

and ending

'

Any idea how to avoid this in general?

btw.. directly calling python to train works without make, I just did training.


#9

I used the hlt_client python script to download replays, and they came through as valid JSON. Perhaps give that a try?


#10

I'm not positive but I seem to recall that was a bug in earlier versions of the client, you might try downloading a fresh copy and see if it works correctly now.


#11

It's worth capturing for the collective memory here, as well, that I've seen an error (different from above) when download replay files have 0 characters. This line in particular will halt: https://github.com/HaliteChallenge/Halite-II/blob/master/airesources/ML-StarterBot-Python/tsmlstarterbot/train.py#L55
However, I suspect that the 0-character length files I had were errors from the hlt_client, possibly an older version.


#12

What command did you use to train via python? Running python3 -m tsmlstarterbot.train ... throws the same error


#13

Any information on how to avoid the binary format of b' in the beginning and ' in the end?