dhall-haskell/dhall-lsp-server/src/Dhall/LSP/Backend/Dhall.hs

170 lines
6.6 KiB
Haskell
Raw Normal View History

dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
module Dhall.LSP.Backend.Dhall (
FileIdentifier,
fileIdentifierFromFilePath,
fileIdentifierFromURI,
hashNormalToCode,
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
WellTyped,
fromWellTyped,
Normal,
fromNormal,
Cache,
emptyCache,
invalidate,
DhallError(..),
parse,
parseWithHeader,
load,
typecheck,
normalize
) where
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
import Dhall.Parser (Src)
import Dhall.Core (Expr)
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
import qualified Dhall.Core as Dhall
import qualified Dhall.Import as Dhall
import qualified Dhall.Parser as Dhall
import qualified Dhall.TypeCheck as Dhall
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
import qualified Data.Graph as Graph
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
import qualified Data.Map.Strict as Map
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
import qualified Data.Set as Set
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
import qualified Network.URI as URI
import qualified Language.Haskell.LSP.Types as LSP.Types
import qualified Data.Text as Text
import Data.List.NonEmpty (NonEmpty((:|)))
import Data.Text (Text)
import Data.Void (Void)
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
import System.FilePath (splitDirectories, takeFileName, takeDirectory)
import Lens.Family (view, set)
import Control.Exception (SomeException, catch)
import Control.Monad.Trans.State.Strict (runStateT)
import Network.URI (URI)
import Data.Bifunctor (first)
-- | A @FileIdentifier@ represents either a local file or a remote url.
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
newtype FileIdentifier = FileIdentifier Dhall.Chained
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | Construct a FileIdentifier from a local file path.
fileIdentifierFromFilePath :: FilePath -> FileIdentifier
fileIdentifierFromFilePath path =
let filename = Text.pack $ takeFileName path
directory = takeDirectory path
components = map Text.pack . reverse . splitDirectories $ directory
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
file = Dhall.File (Dhall.Directory components) filename
in FileIdentifier $ Dhall.chainedFromLocalHere Dhall.Absolute file Dhall.Code
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
-- | Construct a FileIdentifier from a given URI. Supports only "file:" URIs.
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
fileIdentifierFromURI :: URI -> Maybe FileIdentifier
fileIdentifierFromURI uri
| URI.uriScheme uri == "file:" = do
path <- LSP.Types.uriToFilePath . LSP.Types.Uri . Text.pack
$ URI.uriToString id uri ""
return $ fileIdentifierFromFilePath path
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
fileIdentifierFromURI _ = Nothing
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | A well-typed expression.
newtype WellTyped = WellTyped {fromWellTyped :: Expr Src Void}
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | A fully normalised expression.
newtype Normal = Normal {fromNormal :: Expr Src Void}
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
-- An import graph, represented by list of import dependencies.
type ImportGraph = [Dhall.Depends]
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | A cache maps Dhall imports to fully normalised expressions. By reusing
-- caches we can speeds up diagnostics etc. significantly!
data Cache = Cache ImportGraph (Map.Map Dhall.Chained Dhall.ImportSemantics)
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | The initial cache.
emptyCache :: Cache
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
emptyCache = Cache [] Map.empty
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | Invalidate any _unhashed_ imports of the given file. Hashed imports are
-- kept around as per
-- https://github.com/dhall-lang/dhall-lang/blob/master/standard/imports.md.
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
-- Transitively invalidates any imports depending on the changed file.
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
invalidate :: FileIdentifier -> Cache -> Cache
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
invalidate (FileIdentifier chained) (Cache dependencies cache) =
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
Cache dependencies' $ Map.withoutKeys cache invalidImports
where
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
imports = map Dhall.parent dependencies ++ map Dhall.child dependencies
adjacencyLists = foldr
-- add reversed edges to adjacency lists
(\(Dhall.Depends parent child) -> Map.adjust (parent :) child)
-- starting from the discrete graph
(Map.fromList [ (i,[]) | i <- imports])
dependencies
(graph, importFromVertex, vertexFromImport) = Graph.graphFromEdges
[(node, node, neighbours) | (node, neighbours) <- Map.assocs adjacencyLists]
-- compute the reverse dependencies, i.e. the imports reachable in the transposed graph
reachableImports import_ =
map (\(i,_,_) -> i) . map importFromVertex . concat $
do vertex <- vertexFromImport import_
return (Graph.reachable graph vertex)
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
codeImport = Dhall.chainedChangeMode Dhall.Code chained
textImport = Dhall.chainedChangeMode Dhall.RawText chained
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
invalidImports = Set.fromList $ codeImport : reachableImports codeImport
++ textImport : reachableImports textImport
dependencies' = filter (\(Dhall.Depends parent child) -> Set.notMember parent invalidImports
&& Set.notMember child invalidImports) dependencies
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- | A Dhall error. Covers parsing, resolving of imports, typechecking and
-- normalisation.
data DhallError = ErrorInternal SomeException
| ErrorImportSourced (Dhall.SourcedException Dhall.MissingImports)
| ErrorTypecheck (Dhall.TypeError Src Void)
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
| ErrorParse Dhall.ParseError
-- | Parse a Dhall expression.
parse :: Text -> Either DhallError (Expr Src Dhall.Import)
parse = fmap snd . parseWithHeader
-- | Parse a Dhall expression along with its "header", i.e. whitespace and
-- comments prefixing the actual code.
parseWithHeader :: Text -> Either DhallError (Dhall.Header, Expr Src Dhall.Import)
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
parseWithHeader = first ErrorParse . Dhall.exprAndHeaderFromText ""
-- | Resolve all imports in an expression.
load :: FileIdentifier -> Expr Src Dhall.Import -> Cache ->
IO (Either DhallError (Cache, Expr Src Void))
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
load (FileIdentifier chained) expr (Cache graph cache) = do
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
let emptyStatus = Dhall.emptyStatus ""
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
status = -- reuse cache and import graph
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
set Dhall.cache cache .
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
set Dhall.graph graph .
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
-- set "root import"
Preparing `Dhall.Import` for "Semi-semantic" caching (#1113) * Fix misleading comment * Add `Chained` type to capture fully chained imports Until now we used `Import` two mean two different things: - The syntactic construct; e.g. `./a.dhall` corresponds to the following AST: ``` Embed (Import (ImportHashed Nothing (Local Here (Directory ["."]) "a.dhall")) Code) ``` - The physical location the import is pointing to, computed by 'chaining' the syntactical import with the the 'physical' parent import. For example the syntactic import `./a.dhall` might actually refer to the remote file `http://host/directory/a.dhall`. This commit adds a `Chained` newtype on top of `Import` to make this distinction explicit at type level. * Use `HTTPHeaders` alias for binary headers I claim that `HTTPHeaders` is more readable and informative than the unfolded type `(CI ByteString, ByteString)`. * Typecheck and normalise http headers earlier Previously we would typecheck and normalise http headers in `exprFromImport`, i.e. while loading the import. This commit adds the invariant that any headers in 'Chained' imports are already typechecked and normalised, and moves this step into `loadWith` accordingly. This causes a subtle difference in behaviour when importing remote files with headers `as Location`: previously, nonsensical expressions like `http://a using 0 0 as Location` were valid, while they would now cause a type error. * Fix dhall-lsp-server * Fix Dhall.Import API regarding `Chained` imports Do not expose the `Chained` constructor; we don't want external code breaking our invariants! Also further clarifies the comment describing the `Chained` type. * Fix dhall-lsp-server Since we are no longer able to construct `Chained` imports directly we need to export a few additional helper functions from Dhall.Import. Furthermore, since VSCode (and presumably the other editors out there implementing the LSP protocol) does not support opening remote files anyway we can get rid of some complications by dropping support for remote files entirely on the back-end. * Generalise decodeExpression, fixes TODO * Fix tests * Fix benchmarks * Remove Travis cache for `~/.local/bin` * Fix copy-pasted comment Thanks to @Gabriel439 for spotting this! * Add clarifying comment to `toHeaders`
2019-07-17 17:20:48 +02:00
set Dhall.stack (chained :| [])
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
$ emptyStatus
(do (expr', status') <- runStateT (Dhall.loadWith expr) status
dhall-lsp-server: Fix cache to correctly invalidate transitive dependencies (#1069) * Move "Dot" import graph generation to Dhall.Main Previously `Dhall.Import` would generate the import graph in "dot" format while resolving imports. This change simplifies `Dhall.Import` to only keep track of the adjacency list representing the import graph, moving the logic for generating "dot" files to Dhall.Main. This change will allow us to implement proper cache invalidation for `dhall-lsp-server`. * Correctly invalidate transitive dependencies Fixes dhall-lsp-server`s caching behaviour to correctly invalidate cached imports that (possibly indirectly) depend on the changed file. Example: Suppose we have the following three files: {- In A.dhall -} 2 : ./B.dhall {- In B.dhall -} ./C.dhall {- In C.dhall -} Natural Previously, changing C.dhall to `Text` would not cause `A.dhall` to stop type-checking, since the old version of `B.dhall` (which evaluated to `Natural`) would still have been in the cache. This change fixes that behaviour. * Make edges of import graph self-documenting As suggested by @Gabriel439 * Don't cache expressions manually After computing the diagnostics for a given file we added its normal form to the cache, but forgot to add its dependencies to the dependency graph. This bug points out that keeping the import graph consistent manually is probably not a good idea. With this commit we never mess with the import cache manually; this means that files are only cached once they are depended upon by some other file, potentially causing us to duplicate work (but no more than once). * Fix left-overs from previous commit
2019-07-08 12:55:15 +02:00
let cache' = view Dhall.cache status'
graph' = view Dhall.graph status'
return . Right $ (Cache graph' cache', expr'))
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
`catch` (\e -> return . Left $ ErrorImportSourced e)
`catch` (\e -> return . Left $ ErrorInternal e)
-- | Typecheck a fully resolved expression. Returns a certification that the
-- input was well-typed along with its (well-typed) type.
typecheck :: Expr Src Void -> Either DhallError (WellTyped, WellTyped)
dhall-lsp-server: Implement caching (#1040) * Rewriting Dhall.LSP.Backend.Dhall: Implement new API The old "backend" consisted of a random collection of ways to invoke Dhall: - runDhall :: FilePath -> Text -> IO (Expr Src X) - runDhallSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) - loadDhallExprSafe :: FilePath -> Text -> IO (Maybe (Expr Src X)) The new backend exposes a slightly more though-out API. This also lays the foundation for performance improvements in the dhall lsp server via caching. * Reorder code in Dhall.LSP.Backend.Dhall * Remove unused constructor * Rewrite and document Backend.Formatting * Refactor Dhall.LSP.Backend.Linting * Refactor Dhall.LSP.Backend.ToJSON * Adapt Diagnostics backend to the new Dhall API * Remove old Dhall backend API * Implement caching; revamp LSP frontend This commit implements caching of Dhall expressions: we only need to fetch, typecheck and normalise each import once per session, unless they change! This means that `dhall-lsp-server` is now viable for non-trivial Dhall projects, for example probing around in `dhall-nethack` everything feels near-instantaneous once the imports have been resolved. This implementation currently has a bug: we don't invalidate imports transitively, i.e. if A.dhall loads B.dhall and B.dhall changes we do not discard the cached version of A.dhall. This should be reasonably easy to fix given some time with Dhall's import graph. Furthermore, there is some cleaning up left to do: - Fix warnings - Reorganise things in a less ad-hoc way - Make the code a bit prettier * Fix caching of errors * Use `bimap` instead of `first` and `second` * Re-export `Dhall.lint` rather than aliasing Rids us of some boilderplate * Use MVar instead of TVar for server state The main benefit is that we get to use `modifyMVar_` which does updating of the shared state for us (and gracefully handles any uncaught exceptions). * Don't invalidate hashed imports Fixes a misinterpretation on my end of the correct behaviour regarding the caching of imports. Quoting @Gabriel439: > A hashed import is valid indefinitely once it is successfully > resolved, even when the underlying import later becomes broken. That's > why missing sha256:… works so long as the cache has that import cached > (and this behavior is part of the standard). * Cleanup Dhall.LSP.Backend.Dhall a little bit * Add note about fixing cache invalidation * Use TemplateHaskell to generate state lenses * Make types of `typeAt` and `annotateLet` more expressive Both assume the input to be well-typed; by using `WellTyped` rather than `Expr Src X` as the type of their input we can make this explicit. This change exposed a bug (also fixed in this commit) in the type-on-hover functionality: we run `typeAt` only if the input was well-typed _the last time we checked it_ (which was at the last save); this means that if the code changed without being written to disk we would happily try to normalise (in `typeAt`) non-well-typed code... * Fix type of typecheck Typecheck returned the well-typed _type_ of a given expression, while I was assuming it would certify the input to be well-typed. Silly indeed. * Remove `checkDhall` from Dhall.Backend.Diagnostics Removes the left-over stub from the change to the new Dhall backend. * Update comments and remove TODO note * Remove superfluous parentheses * Simplify MonadState code via lens combinators * Use `guard` instead of matching on True * Remove more superfluous parentheses
2019-07-01 19:30:32 +02:00
typecheck expr = case Dhall.typeOf expr of
Left err -> Left $ ErrorTypecheck err
Right typ -> Right (WellTyped expr, WellTyped typ)
-- | Normalise a well-typed expression.
normalize :: WellTyped -> Normal
normalize (WellTyped expr) = Normal $ Dhall.normalize expr
-- | Given a normal expression compute the hash (using the default standard
-- version) of its alpha-normal form. Returns the hash in the format used in
-- Dhall's hash annotations (prefixed by "sha256:" and base-64 encoded).
hashNormalToCode :: Normal -> Text
hashNormalToCode (Normal expr) =
Improve encoding/decoding speed (#1500) ... by not going through a `Term` intermediate This gives a ~28% performance in decoding improvement, which means that cache looks are not faster. Here are the new decoding benchmarks before and after this change: Before: ``` benchmarked Issue #108/Binary time 266.5 μs (265.7 μs .. 267.4 μs) 1.000 R² (1.000 R² .. 1.000 R²) mean 266.3 μs (265.6 μs .. 267.1 μs) std dev 2.418 μs (1.891 μs .. 3.436 μs) benchmarking Kubernetes/Binary ... took 36.94 s, total 56 iterations benchmarked Kubernetes/Binary time 641.3 ms (623.0 ms .. 655.4 ms) 0.999 R² (0.997 R² .. 1.000 R²) mean 679.7 ms (665.5 ms .. 702.6 ms) std dev 29.48 ms (14.15 ms .. 39.05 ms) ``` After: ``` benchmarked Issue #108/Binary time 282.2 μs (279.6 μs .. 284.7 μs) 1.000 R² (0.999 R² .. 1.000 R²) mean 281.9 μs (280.7 μs .. 287.7 μs) std dev 7.089 μs (2.550 μs .. 15.44 μs) variance introduced by outliers: 11% (moderately inflated) benchmarking Kubernetes/Binary ... took 27.57 s, total 56 iterations benchmarked Kubernetes/Binary time 499.1 ms (488.1 ms .. 506.6 ms) 0.999 R² (0.998 R² .. 1.000 R²) mean 498.9 ms (494.4 ms .. 503.9 ms) std dev 8.539 ms (6.236 ms .. 12.56 ms) ``` There's a slight performance regression for the decoding microbenchmark, but in practice my testing on real examples matches performance improvements seen in the larger benchmark based on an example cache product from `dhall-kubernetes`. Note that is a breaking change because: * There is no longer a `FromTerm` nor `ToTerm` class. Now we use the `Serialise` class and `{encode,decode}Expression` now work on `ByteString`s instead of `Term`s * I further narrowed the types of several encoding/decoding utilites to expect a `Void` for the first type parameter of `Expr` * This is a regression with respect to stripping 55799 CBOR tags, mainly because properly handling the tags at every possible point in the syntax tree would considerably complicate the code
2019-11-01 04:05:22 +01:00
Dhall.hashExpressionToCode (Dhall.denote alphaNormal)
where alphaNormal = Dhall.alphaNormalize expr