* plotting|tests: Extend the lifetime of plot data cache entries
* plotting|tests: Add `prover` and `farmer_public_key` to the cache
* plotting: Missing type hints in `Cache` and `CacheEntry`
* setup: Bump `chiapos` to 1.0.10
* plotting: Move cache classes into `chia.plotting.cache`
* plotting: Introduce `CacheEntry.from_disk_prover`
* Make `Cache` a dataclass
* `staticmethod` -> `classmethod`
* Store cache data as `bytes` in `DiskCache`
* protocols|server: Define new harvester plot refreshing protocol messages
* protocols: Bump `protocol_version` to `0.0.34`
* tests: Introduce `setup_farmer_multi_harvester`
Allows to run a test setup with 1 farmer and mutiple harvesters.
* plotting: Add an initial plot loading indication to `PlotManager`
* plotting|tests: Don't add removed duplicates to `total_result.removed`
`PlotRefreshResult.removed` should only contain plots that were loaded
properly before they were removed. It shouldn't contain e.g. removed
duplicates or invalid plots since those are synced in an extra sync step
and not as diff but as whole list every time.
* harvester: Reset `PlotManager` on shutdown
* plot_sync: Implement plot sync protocol
* farmer|harvester: Integrate and enable plot sync
* tests: Implement tests for the plot sync protocol
* farmer|tests: Drop obsolete harvester caching code
* setup: Add `chia.plot_sync` to packages
* plot_sync: Type hints in `DeltaType`
* plot_sync: Drop parameters in `super()` calls
* plot_sync: Introduce `send_response` helper in `Receiver._process`
* plot_sync: Add some parentheses
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* plot_sync: Additional hint for a `Receiver.process_path_list` parameter
* plot_sync: Force named parameters in `Receiver.process_path_list`
* test: Fix fixtures after rebase
* tests: Fix sorting after rebase
* tests: Return type hint for `plot_sync_setup`
* tests: Rename `WSChiaConnection` and move it in the outer scope
* tests|plot_sync: More type hints
* tests: Rework some delta tests
* tests: Drop a `range` and iterate over the list directly
* tests: Use the proper flags to overwrite
* test: More missing duplicates tests
* tests: Drop `ExpectedResult.reset`
* tests: Reduce some asserts
* tests: Add messages to some `assert False` statements
* tests: Introduce `ErrorSimulation` enum in `test_sync_simulated.py`
* tests: Use `secrects` instead of `Crypto.Random`
* Fixes after rebase
* Import from `typing_extensions` to support python 3.7
* Drop task name to support python 3.7
* Introduce `Sender.syncing`, `Sender.connected` and a log about the task
* Add `tests/plot_sync/config.py`
* Align the multi harvester fixture with what we do in other places
* Update the workflows
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Add coverage (without collection)
* Separate test_block_compression() to avoid coverage-related hangs
* Revert "Separate test_block_compression() to avoid coverage-related hangs"
This reverts commit ebad3d0017.
* multiprocessing.set_start_method("spawn")
* multiprocessing.set_start_method() in conftest.py
* hand hold cc wallet tests
* lint
* spawn for running chia as well
* handle already set start method case
* a bit more timeout for test_multiple_writers
* more timeout for test_writer_lock_blocked_by_readers
* 45 minute tieout for tests/pools/
* 45 minute tieout for tests/pools/
* some more hand holding sleeps
* report coverage in each workflow
only really useful to make sure it is capturing something
* oops
* complete the job name and the JOB_NAME
* better coverage result file names
* reset worker process titles
* rebuild workflows
* rebuild workflows
* black
* black
* rebuild workflows
* push timeouts
* actually include the updated workflows...
* push more workflow timeouts
* parallel=True
* rebuild workflows
* Enable clvm_tools_rs by default
* Re-add clvm_tools dep for now as it provides python idioms for interacting with clvm data
* Take lint formatting
* Adam: Try making this non-parallel
* Try fix for threading issue in tests
* Test whether turning off parallel runs causes things to work (temp)
* Test use of temp files in clvm_tools_rs as a candidate solution for atomic replacement of hex output
* Use proper git+https url scheme (oops)
* Update to candidate 0.1.6 so we can test
* Revert version bump to re-test
* Test whether we can re-enable parallelism
* Attempt to mitigate concurrent test running: return own conception of the compiled output. This will work if the failing path is downstream of recompilation
* fix path to hex file
* Probe for source of 0-length data
* Further exploration: more assertions, hopefully to trigger in the test on ci
* Do an even more paranoid check to verify that we observe a file whose filesystem reported size is much smaller than expected
* Try a heavier handed approach, using heavyweight lockfiles on the filesystem
* Import a simple lockfile implementation and use it to enforce mutual exclusion. Simplify it and remove the unwanted os traffic
* Take lint, precommit advice, bump to clvm_tools_rs 0.1.6 now that it's released
* Fix lint
* While i was working on this, -n auto was on the command line so i think this didn't actually do anything, but reverting my change just in case
* Lint
* label the hashes re: pr
* Add a lock.py for spot exclusivity using the filesystem (re: adam in the pr) and a convenience wrapper that hides the details
* Formatting warning
* Ensure type info is present and do the obvious return of the inner function's result
* Use double quotes (lint)
* Properly balance blank lines
* Lint: alphabetize imports
* One line is required here (lint)
* Remove unnecessary assignment
* Add configuration locking
Extracted from https://github.com/Chia-Network/chia-blockchain/pull/10631
* note that fasteners will likely be replaced by filelock
* Fix test_multiple_writers on macOS
* create_all_ssl() doesn't need to be inside the config access lock
* add warnings about not using async within get_config_lock() get lock contexts
* no need to pre-touch the lock file
* .yaml.lock instead of just .lock
* test_multiple_writers() is sync
* Revert "add warnings about not using async within get_config_lock() get lock contexts"
This reverts commit 681af3835b.
* reduce lock context size in chia_init()
* use an exit stack in load_config()
* avoid config existence precheck
* only lock around the read in load_config()
* do not raise e, just raise
* tidy new imports
* fix queue empty check in test_config.py
* remove commented out code in test_config.py
* remove unused import
Co-authored-by: Jeff Cruikshank <jeff@chia.net>
* Use bt fixture
* rebase
* Use local_hostname
* Fix test_json (inheritance from unittest.TestCase)
* Use correct BlockTools fixture for test_simulation
* Pass bt fixture into cost calculation tests
* flake8
* Add missing parameters to test functions
* Fix from rebase issues
* Remove set_shared_instance
* Update comment
* Remove unneeded comments
* Remove unused code
* Remove unused code, run `multiprocessing.set_start_method("spawn")` at correct time.
* Revert unrelated change
* Set daemon_port. Teardown services in correct order.
BIG thanks to Mariano Sorgente for debugging help.
* Add back type signature - rebase issue
* Apply review fixes from Jeff
* Document why we need a later pytest-asyncio version
* Correct type for _configure_legacy_backend
* See what's going on during CI mypy run
* github workflows
* mypy typing
* Remove legacy Keyring create method
* Start daemon first
* Shutdown daemon coroutine properly
* Remove un-needed daemon_port argument
* Set chia-blockchain-gui to hash in main
* Remove connect_to_daemon_port
* Remove code that set "daemon_port" before calling `setup_daemon`
* Remove self_hostname fixture and extra self_hostname global
* Fix two test files that were not importing self_hostname
* self_hostname fixture
* Remove more unused test code
* Simplify fixture
* pre-commit: Add a new hook to run `isort`
* contributing: Add hint about `isort`
* add isort to dev deps, ignore existing .py files, use black profile
* long list to lines not comma delimited
* isort: Update and sort ignore list to match latest `main`
* add bash command line to generate isort extend skip list (#3)
* add bash command line to generate extend skip list
* tidy
* isort: More files to ignore after rebase
* tests: Fix `test_wallet_user_store.py` after rebase
* Some fixes after rebase
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Add crawler RPC server
* Generate private keypair for crawler
* Bring over cleanup changes from the last closed PR
* Update the crawler RPC information to be its own subsection within seeder
* Add sleep before crawling to let the daemon connection get set up
* Wait for the actual callback to not be None, instead of just a random sleep interval
* Rework crawler/dns seeder to use the daemon + normal chia start process rather than the old system intended for the standalone repo
* Update configure testnet to work with seeder config
* Add back the crawler/seeder options from the standalone version
* Remove the check for none/sleep. Not needed when this is started by the daemon
* Add real data to the get_peer_counts endpoint
* Lint
* Fix calls to configure from init
* Turns out we still might sometimes move too quick before daemon/state changed callback is ready
* Add peer counts in the state_changed callback method
* Add a setting for peer_connect_timeout in the seeder: section so we can control it just for crawler
* start_seeder
* Pass config/root_path to the DNSServer so it can also use the configured crawler DB Path
* change in () instead of if/or
* Remove unnecessary return
* introduce restrictions on generators at a specific height. disallow division on negative numbers and disallow redundant leading zeros on integer condition arguments (produced by a generator)
* use SOFT_FORK_HEIGHT constant
* there is no need to specify height when validating block in block_creation
* Update tests/core/full_node/test_mempool.py
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* wallet changes from pac
* cat changes
* pool tests
* pooling tests passing
* offers
* lint
* mempool_mode
* black
* linting
* workflow files
* flake8
* more cleanup
* renamed
* remove obsolete test, don't cast announcement
* memos are not only bytes32
* trade renames
* fix rpcs, block_record
* wallet rpc, recompile settlement clvm
* key derivation
* clvm tests
* lgtm issues and wallet peers
* stash
* rename
* mypy linting
* flake8
* bad initializer
* flaky tests
* Make CAT wallets only create on verified hints (#9651)
* fix clvm tests
* return to log lvl warn
* check puzzle unhardened
* public key, not bytes. api caching change
* precommit changes
* remove unused import
* mypy ci file, tests
* ensure balance before creating a tx
* Remove CAT logic from full node test (#9741)
* Add confirmations and sleeps for wallet (#9742)
* use pool executor
* rever merge mistakes/cleanup
* Fix trade test flakiness (#9751)
* remove precommit
* older version of black
* lint only in super linter
* Make announcements in RPC be objects instead of bytes (#9752)
* Make announcements in RPC be objects instead of bytes
* Lint
* misc hint'ish cleanup (#9753)
* misc hint'ish cleanup
* unremove some ci bits
* Use main cached_bls.py
* Fix bad merge in main_pac (#9774)
* Fix bad merge at 71da0487b9
* Remove unused ignores
* more unused ignores
* Fix bad merge at 3b143e7050
* One more byte32.from_hexstr
* Remove obsolete test
* remove commented out
* remove duplicate payment object
* remove long sync
* remove unused test, noise
* memos type
* bytes32
* make it clear it's a single state at a time
* copy over asset ids from pacr
* file endl linter
* Update chia/server/ws_connection.py
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: Matt Hauff <quexington@gmail.com>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* initial hack
* crawler
* add pytz
* Checkpoint.
* Catch some bugs.
* Localhost dig working.
* Checkpoint: return only high quality nodes.
* Statistics.
* Try improving finding reliable nodes.
* Bug.
* Move db to memory.
* Timestamp in the last 5 days.
* Increase crawl parameters, 180+ connections per sec.
* Bug.
* Optimize for DNS traffic.
* Prepare for hosting.
* Minimum height.
* Typo.
* Try catch everything.
* dnslib.
* Add db, format code.
* nits.
* No connections for the dns server.
* Rename src -> chia
* Fix some issues with v1.1
* Crawler task pool.
* Optimize closing connections.
* Split crawler and dns server.
* Install instructions.
* Catch startup bug.
* Try a big timeout for lock aquire.
* lint.
* Lint.
* Initial commit extended stats.
* Simplify code.
* Config.
* Correct stats.
* Be more restrictive in crawling.
* Attempt to fix stats bug.
* Add other peers port to config.
* Update README for the config.
* Simplify crawl task.
* Fix bug on restarts.
* Prevent log spamming.
* More spam prevention.
* Fix bug.
* Ipv6 (#1)
* Enable ipv6.
* Fix bug.
* Use numeric codes for QTYPE.
* ANY working.
* More spam prevention.
* Try to improve IPv6 selection.
* Log IPv6 available.
* Try to crawl more aggresive for v6.
* rename dns.py to crawler_dns.py so it doesn't conflict with imported package names
* Remove pytz package off dependencies
* Tidy-up ws_connection.py
* Fix spelling
* Reinstate chia-blockchain readme, with additional lines pertaining to the DNS introducer & crawler
* More detailed info in the README wrt Chia Seeder
* Nit
* More memetic naming of Chia Seeder
* Nit
* Add entry points
* Add entry in packages
* Patch some methods on the upstream server
* Update peer record fields
* Standard library imports first
* Crawler API check
* Reconcile crawl store
* Account for crawler_db_path in config
* Await crawl store load DB and load reliable peers
* Updates to crawler
* Rename to dns_server
* Crawler-specific overrides for the chia server
* Edit comment
* Undo changes to ChiaServer in view of crawler-specific overrides introduced in previous commit
* Nit
* Update service groups
* Expand name maps, mostly
* Fix the init config
* Remove unused import
* total_records unused at this stage
* Remove ios_reliable in peer_reliability table
* Remove row[20] entry
* Split overly long line
* Fix
* Type hint for ns_records
* Reconcile mismatch btw type int and uint64
* Type annotations in crawler
* Check whether crawl store is set
* Remove upnp_list
* Lint
* Chia Seeder CLI
* Lint
* Two white spaces
* 3rd party package import
* Cleaner way to handle overrides for ChiaServer method
* Address linter warnings
* Rename
* Nits
* Fix
* Change port #
* Most chia_seeder commands up and running
* Rename
* Progress of sorts
* Fix
* Improve legibility
* Fix naming
* Fix setup.py
* Lint
* None -> ''
* Remove whitespace
* Rename
* Log ipv6 better. (#9227)
* Log ipv6 better.
* Lint.
* -
* Undo GUI changes
* Another attempt
* GUI changes
Co-authored-by: Yostra <straya@chia.net>
Co-authored-by: Florin Chirica <fchirica96@gmail.com>
Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
* Install .[dev] in CI tests
* only install pytest-monitor on linux, as before
* add -d to install.sh for dev extras
* handle invalid options to install.sh
* rebuild workflows
* Serialize/deserialize peer data alongside existing sqlite implementation (to be removed)
* Simplified AddressManagerStore. No longer uses sqlite and is no longer async.
* Removed aiosqlite usage from AddressManagerStore.
Added PeerStoreResolver class to determine the appropriate location for "peers.dat"
Updated initial-config.yaml to include "peers_file_path" default, replacing "peer_db_path" (similar change for "wallet_peers_path")
* Minor comment changes/additions
* Added migration from sqlite peer db.
Made AddressManagerStore's serialization async as it was right at the edge of blocking for too long.
* Minor tweaks to checking for migration
* Removed AddressManagerSQLiteStore class scaffolding
* makePeerDataSerialization now returns bytes instead of a PeerDataSerialization object
* Async file I/O for write_file_async using aiofiles
Added more tests
* Separate out the synchronous part of move_file
* Renamed write_file to files since we're opening up the capabilities a bit
* Update references to write_file
* Renamed test_write_file to test_files
* Tests covering move_file and move_file_async
* Minor refinements to behavior and tests
* Use aiofiles for reading peers.dat
* Added missing mypy typing info for aiofiles. Also added types-PyYAML to dev_dependencies so that `mypy chia tests` doesn't require running with --install-types
* Add types-aiofiles to the linting workflow
* Directory perms can now be passed into write_file_async.
Added an explicit f.flush() followed by os.fsync() after writing the temp file contents.
* Initial commit add plotters.
* Lint.
* Add progress for bladebit.
* Address some review comments.
* Oops...
* Add install option.
* Change chiapos to fit the old standard.
* Update chia/plotters/bladebit.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/install_plotter.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Lint.
* Remove farmerkey as required.
* Chia plotters chiapos works with no arguments.
* Added get_plotters RPC to support the GUI (#8530)
* Added 'get_plotters' daemon RPC. Probes installed/available plotters on behalf of the GUI.
* Linter fix
* Minor type tweak
* Tweaks.
* Run with default arguments all plotters.
* Fix bug.
* Update chia/plotters/bladebit.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Change bladebit repo.
* Update chia/plotters/bladebit.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Update chia/plotters/plotters.py
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
* Re-added the connect-to-daemon hidden option
* Use connect_to_daemon value for madmax and bladebit plot key resolution.
* Updated --tmp_dir, --tmp_dir2, --final_dir options to match old options from chia plots create
* Add CONNECT_TO_DAEMON as a valid option for madmax/bladebit
* Thread multiplier should be an int
* Passing params for madmax to start_plotting. Still needs cleanup/refactoring.
* Update chia/plotters/bladebit.py
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Update chia/plotters/madmax.py
Co-authored-by: Kyle Altendorf <sda@fstab.net>
* Filename option -z.
* Factor out calling the plotter.
* First attempt refactor install scripts.
* Switch to exec.
* Attempt to fix mypy warning.
* Remove filename.
* Increase RLIMIT_NOFILE for madmax on non-Windows platforms
* Add trailing path separator to madmax tmpdir/tmp2dir/finaldir arguments (required by madmax)
* Fixes to support madmax plotting from the GUI.
Writing output from the plotters now includes a flush to ensure the plotter log (used by the GUI) is updated frequently.
* Handle madmax's tmptoggle option internally when plotting with the GUI
* Formatting and linter fix
* Fixed the -i option for bladebit
* Construct BladeBit plotting options
* Cleanup code for building plotter command line options
* Added a post-processing step after each plotting job completes. Adds the final_dir plot directory as necessary.
* Fix plotter root path
* Reverting prior checkin. Need to figure out how to handle CHIA_ROOT being overridden
* BladeBit support for Windows
* BladeBit's --memory-json option is used to check memory requirements
* Madmax Windows support
* Plotters directory is now under CHIA_ROOT
* Madmax version detection
* BladeBit will default to 0 threads. BladeBit will max-out available threads with this configuration.
* LGTM fixes
* Module definition for chia.plotters to resolve mypy issues with chiapos (package vs chiapos.py)
* Updated BladeBit build script to account for 1.2.0 changes.
Replaced remaining subprocess.run calls with run_command.
Use BladeBit's reported memory requirement instead of hardcoded value.
* Show a disclaimer when using thirdparty plotter
* Test adding mac madmax plotter to the installers
* Get latest madmax from the latest GH release
* Fix bad var name
* m1 madmax
* Add linux/linux arm
* pip install -e for arm installer, so that its consistent with the other platforms when looking for additional files
* madmax + windows
* Get madmax with Invoke-WebRequest
* Use the correct windows slashes
* add madmax to the list of windows binaries
* Check if madmax exists on windows install and move it to site packages if it does
* Make sure windows has .exe extension for madmax
* Update azure to get latest version of madmax from GH releases
* Bladebit for linux/linux arm
* Fix error with binaries.extend
* Bundle bladebit + windows
* Fix download url for bladebit
* Check for bladebit in windows script. move to correct directory if it exists
* Detect and use packaged plotters
* Removed unnecessary import
* Updating the branch to use chiaplotters_gui for installer verification
* Removed a change that was intended for debugging only
* Remove disclaimer
* Updated for new madmax plotter with k33, k34 support.
Updated chia-blockchain-gui submodule
* Fixed typo
* Package the chia_plot_k34 executable
* Boink
* Revert "Boink"
This reverts commit 8d13c07110.
* Additional chia_plot_k34 spots that I missed
* pyinstaller.spec fix for chia_plot_k34
* Windows installer fix for chia_plot_k34.exe
* Restoring chia-blockchain-gui submodule to 047ce16 (as in main)
* Update to chiapos 1.0.6
Co-authored-by: Jeff Cruikshank <paninaro@gmail.com>
Co-authored-by: Jeff Cruikshank <jeff@chia.net>
Co-authored-by: Kyle Altendorf <sda@fstab.net>
Co-authored-by: Chris Marslender <chrismarslender@gmail.com>
Co-authored-by: Earle Lowe <30607889+emlowe@users.noreply.github.com>
* github: Add `pre-commit.yml`
This runs `pre-commit` as github action for all PRs and on pushes to
`main`.
* github: Drop trailing whitespaces in `stale_issue.yml`
* setup: Ignore type errors for `setup` call
