You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 

497 lines
16 KiB

  1. // Copyright 2016 Google LLC
  2. //
  3. // Licensed under the Apache License, Version 2.0 (the "License");
  4. // you may not use this file except in compliance with the License.
  5. // You may obtain a copy of the License at
  6. //
  7. // http://www.apache.org/licenses/LICENSE-2.0
  8. //
  9. // Unless required by applicable law or agreed to in writing, software
  10. // distributed under the License is distributed on an "AS IS" BASIS,
  11. // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  12. // See the License for the specific language governing permissions and
  13. // limitations under the License.
  14. /*
  15. Package datastore provides a client for Google Cloud Datastore.
  16. See https://godoc.org/cloud.google.com/go for authentication, timeouts,
  17. connection pooling and similar aspects of this package.
  18. Basic Operations
  19. Entities are the unit of storage and are associated with a key. A key
  20. consists of an optional parent key, a string application ID, a string kind
  21. (also known as an entity type), and either a StringID or an IntID. A
  22. StringID is also known as an entity name or key name.
  23. It is valid to create a key with a zero StringID and a zero IntID; this is
  24. called an incomplete key, and does not refer to any saved entity. Putting an
  25. entity into the datastore under an incomplete key will cause a unique key
  26. to be generated for that entity, with a non-zero IntID.
  27. An entity's contents are a mapping from case-sensitive field names to values.
  28. Valid value types are:
  29. - Signed integers (int, int8, int16, int32 and int64)
  30. - bool
  31. - string
  32. - float32 and float64
  33. - []byte (up to 1 megabyte in length)
  34. - Any type whose underlying type is one of the above predeclared types
  35. - *Key
  36. - GeoPoint
  37. - time.Time (stored with microsecond precision, retrieved as local time)
  38. - Structs whose fields are all valid value types
  39. - Pointers to structs whose fields are all valid value types
  40. - Slices of any of the above
  41. - Pointers to a signed integer, bool, string, float32, or float64
  42. Slices of structs are valid, as are structs that contain slices.
  43. The Get and Put functions load and save an entity's contents. An entity's
  44. contents are typically represented by a struct pointer.
  45. Example code:
  46. type Entity struct {
  47. Value string
  48. }
  49. func main() {
  50. ctx := context.Background()
  51. // Create a datastore client. In a typical application, you would create
  52. // a single client which is reused for every datastore operation.
  53. dsClient, err := datastore.NewClient(ctx, "my-project")
  54. if err != nil {
  55. // Handle error.
  56. }
  57. k := datastore.NameKey("Entity", "stringID", nil)
  58. e := new(Entity)
  59. if err := dsClient.Get(ctx, k, e); err != nil {
  60. // Handle error.
  61. }
  62. old := e.Value
  63. e.Value = "Hello World!"
  64. if _, err := dsClient.Put(ctx, k, e); err != nil {
  65. // Handle error.
  66. }
  67. fmt.Printf("Updated value from %q to %q\n", old, e.Value)
  68. }
  69. GetMulti, PutMulti and DeleteMulti are batch versions of the Get, Put and
  70. Delete functions. They take a []*Key instead of a *Key, and may return a
  71. datastore.MultiError when encountering partial failure.
  72. Mutate generalizes PutMulti and DeleteMulti to a sequence of any Datastore mutations.
  73. It takes a series of mutations created with NewInsert, NewUpdate, NewUpsert and
  74. NewDelete and applies them atomically.
  75. Properties
  76. An entity's contents can be represented by a variety of types. These are
  77. typically struct pointers, but can also be any type that implements the
  78. PropertyLoadSaver interface. If using a struct pointer, you do not have to
  79. explicitly implement the PropertyLoadSaver interface; the datastore will
  80. automatically convert via reflection. If a struct pointer does implement
  81. PropertyLoadSaver then those methods will be used in preference to the default
  82. behavior for struct pointers. Struct pointers are more strongly typed and are
  83. easier to use; PropertyLoadSavers are more flexible.
  84. The actual types passed do not have to match between Get and Put calls or even
  85. across different calls to datastore. It is valid to put a *PropertyList and
  86. get that same entity as a *myStruct, or put a *myStruct0 and get a *myStruct1.
  87. Conceptually, any entity is saved as a sequence of properties, and is loaded
  88. into the destination value on a property-by-property basis. When loading into
  89. a struct pointer, an entity that cannot be completely represented (such as a
  90. missing field) will result in an ErrFieldMismatch error but it is up to the
  91. caller whether this error is fatal, recoverable or ignorable.
  92. By default, for struct pointers, all properties are potentially indexed, and
  93. the property name is the same as the field name (and hence must start with an
  94. upper case letter).
  95. Fields may have a `datastore:"name,options"` tag. The tag name is the
  96. property name, which must be one or more valid Go identifiers joined by ".",
  97. but may start with a lower case letter. An empty tag name means to just use the
  98. field name. A "-" tag name means that the datastore will ignore that field.
  99. The only valid options are "omitempty", "noindex" and "flatten".
  100. If the options include "omitempty" and the value of the field is a zero value,
  101. then the field will be omitted on Save. Zero values are best defined in the
  102. golang spec (https://golang.org/ref/spec#The_zero_value). Struct field values
  103. will never be empty, except for nil pointers.
  104. If options include "noindex" then the field will not be indexed. All fields
  105. are indexed by default. Strings or byte slices longer than 1500 bytes cannot
  106. be indexed; fields used to store long strings and byte slices must be tagged
  107. with "noindex" or they will cause Put operations to fail.
  108. For a nested struct field, the options may also include "flatten". This
  109. indicates that the immediate fields and any nested substruct fields of the
  110. nested struct should be flattened. See below for examples.
  111. To use multiple options together, separate them by a comma.
  112. The order does not matter.
  113. If the options is "" then the comma may be omitted.
  114. Example code:
  115. // A and B are renamed to a and b.
  116. // A, C and J are not indexed.
  117. // D's tag is equivalent to having no tag at all (E).
  118. // I is ignored entirely by the datastore.
  119. // J has tag information for both the datastore and json packages.
  120. type TaggedStruct struct {
  121. A int `datastore:"a,noindex"`
  122. B int `datastore:"b"`
  123. C int `datastore:",noindex"`
  124. D int `datastore:""`
  125. E int
  126. I int `datastore:"-"`
  127. J int `datastore:",noindex" json:"j"`
  128. }
  129. Slice Fields
  130. A field of slice type corresponds to a Datastore array property, except for []byte, which corresponds
  131. to a Datastore blob.
  132. Zero-length slice fields are not saved. Slice fields of length 1 or greater are saved
  133. as Datastore arrays. When a zero-length Datastore array is loaded into a slice field,
  134. the slice field remains unchanged.
  135. If a non-array value is loaded into a slice field, the result will be a slice with
  136. one element, containing the value.
  137. Loading Nulls
  138. Loading a Datastore Null into a basic type (int, float, etc.) results in a zero value.
  139. Loading a Null into a slice of basic type results in a slice of size 1 containing the zero value.
  140. Loading a Null into a pointer field results in nil.
  141. Loading a Null into a field of struct type is an error.
  142. Pointer Fields
  143. A struct field can be a pointer to a signed integer, floating-point number, string or
  144. bool. Putting a non-nil pointer will store its dereferenced value. Putting a nil
  145. pointer will store a Datastore Null property, unless the field is marked omitempty,
  146. in which case no property will be stored.
  147. Loading a Null into a pointer field sets the pointer to nil. Loading any other value
  148. allocates new storage with the value, and sets the field to point to it.
  149. Key Field
  150. If the struct contains a *datastore.Key field tagged with the name "__key__",
  151. its value will be ignored on Put. When reading the Entity back into the Go struct,
  152. the field will be populated with the *datastore.Key value used to query for
  153. the Entity.
  154. Example code:
  155. type MyEntity struct {
  156. A int
  157. K *datastore.Key `datastore:"__key__"`
  158. }
  159. func main() {
  160. ctx := context.Background()
  161. dsClient, err := datastore.NewClient(ctx, "my-project")
  162. if err != nil {
  163. // Handle error.
  164. }
  165. k := datastore.NameKey("Entity", "stringID", nil)
  166. e := MyEntity{A: 12}
  167. if _, err := dsClient.Put(ctx, k, &e); err != nil {
  168. // Handle error.
  169. }
  170. var entities []MyEntity
  171. q := datastore.NewQuery("Entity").Filter("A =", 12).Limit(1)
  172. if _, err := dsClient.GetAll(ctx, q, &entities); err != nil {
  173. // Handle error
  174. }
  175. log.Println(entities[0])
  176. // Prints {12 /Entity,stringID}
  177. }
  178. Structured Properties
  179. If the struct pointed to contains other structs, then the nested or embedded
  180. structs are themselves saved as Entity values. For example, given these definitions:
  181. type Inner struct {
  182. W int32
  183. X string
  184. }
  185. type Outer struct {
  186. I Inner
  187. }
  188. then an Outer would have one property, Inner, encoded as an Entity value.
  189. If an outer struct is tagged "noindex" then all of its implicit flattened
  190. fields are effectively "noindex".
  191. If the Inner struct contains a *Key field with the name "__key__", like so:
  192. type Inner struct {
  193. W int32
  194. X string
  195. K *datastore.Key `datastore:"__key__"`
  196. }
  197. type Outer struct {
  198. I Inner
  199. }
  200. then the value of K will be used as the Key for Inner, represented
  201. as an Entity value in datastore.
  202. If any nested struct fields should be flattened, instead of encoded as
  203. Entity values, the nested struct field should be tagged with the "flatten"
  204. option. For example, given the following:
  205. type Inner1 struct {
  206. W int32
  207. X string
  208. }
  209. type Inner2 struct {
  210. Y float64
  211. }
  212. type Inner3 struct {
  213. Z bool
  214. }
  215. type Inner4 struct {
  216. WW int
  217. }
  218. type Inner5 struct {
  219. X Inner4
  220. }
  221. type Outer struct {
  222. A int16
  223. I []Inner1 `datastore:",flatten"`
  224. J Inner2 `datastore:",flatten"`
  225. K Inner5 `datastore:",flatten"`
  226. Inner3 `datastore:",flatten"`
  227. }
  228. an Outer's properties would be equivalent to those of:
  229. type OuterEquivalent struct {
  230. A int16
  231. IDotW []int32 `datastore:"I.W"`
  232. IDotX []string `datastore:"I.X"`
  233. JDotY float64 `datastore:"J.Y"`
  234. KDotXDotWW int `datastore:"K.X.WW"`
  235. Z bool
  236. }
  237. Note that the "flatten" option cannot be used for Entity value fields.
  238. The server will reject any dotted field names for an Entity value.
  239. The PropertyLoadSaver Interface
  240. An entity's contents can also be represented by any type that implements the
  241. PropertyLoadSaver interface. This type may be a struct pointer, but it does
  242. not have to be. The datastore package will call Load when getting the entity's
  243. contents, and Save when putting the entity's contents.
  244. Possible uses include deriving non-stored fields, verifying fields, or indexing
  245. a field only if its value is positive.
  246. Example code:
  247. type CustomPropsExample struct {
  248. I, J int
  249. // Sum is not stored, but should always be equal to I + J.
  250. Sum int `datastore:"-"`
  251. }
  252. func (x *CustomPropsExample) Load(ps []datastore.Property) error {
  253. // Load I and J as usual.
  254. if err := datastore.LoadStruct(x, ps); err != nil {
  255. return err
  256. }
  257. // Derive the Sum field.
  258. x.Sum = x.I + x.J
  259. return nil
  260. }
  261. func (x *CustomPropsExample) Save() ([]datastore.Property, error) {
  262. // Validate the Sum field.
  263. if x.Sum != x.I + x.J {
  264. return nil, errors.New("CustomPropsExample has inconsistent sum")
  265. }
  266. // Save I and J as usual. The code below is equivalent to calling
  267. // "return datastore.SaveStruct(x)", but is done manually for
  268. // demonstration purposes.
  269. return []datastore.Property{
  270. {
  271. Name: "I",
  272. Value: int64(x.I),
  273. },
  274. {
  275. Name: "J",
  276. Value: int64(x.J),
  277. },
  278. }, nil
  279. }
  280. The *PropertyList type implements PropertyLoadSaver, and can therefore hold an
  281. arbitrary entity's contents.
  282. The KeyLoader Interface
  283. If a type implements the PropertyLoadSaver interface, it may
  284. also want to implement the KeyLoader interface.
  285. The KeyLoader interface exists to allow implementations of PropertyLoadSaver
  286. to also load an Entity's Key into the Go type. This type may be a struct
  287. pointer, but it does not have to be. The datastore package will call LoadKey
  288. when getting the entity's contents, after calling Load.
  289. Example code:
  290. type WithKeyExample struct {
  291. I int
  292. Key *datastore.Key
  293. }
  294. func (x *WithKeyExample) LoadKey(k *datastore.Key) error {
  295. x.Key = k
  296. return nil
  297. }
  298. func (x *WithKeyExample) Load(ps []datastore.Property) error {
  299. // Load I as usual.
  300. return datastore.LoadStruct(x, ps)
  301. }
  302. func (x *WithKeyExample) Save() ([]datastore.Property, error) {
  303. // Save I as usual.
  304. return datastore.SaveStruct(x)
  305. }
  306. To load a Key into a struct which does not implement the PropertyLoadSaver
  307. interface, see the "Key Field" section above.
  308. Queries
  309. Queries retrieve entities based on their properties or key's ancestry. Running
  310. a query yields an iterator of results: either keys or (key, entity) pairs.
  311. Queries are re-usable and it is safe to call Query.Run from concurrent
  312. goroutines. Iterators are not safe for concurrent use.
  313. Queries are immutable, and are either created by calling NewQuery, or derived
  314. from an existing query by calling a method like Filter or Order that returns a
  315. new query value. A query is typically constructed by calling NewQuery followed
  316. by a chain of zero or more such methods. These methods are:
  317. - Ancestor and Filter constrain the entities returned by running a query.
  318. - Order affects the order in which they are returned.
  319. - Project constrains the fields returned.
  320. - Distinct de-duplicates projected entities.
  321. - KeysOnly makes the iterator return only keys, not (key, entity) pairs.
  322. - Start, End, Offset and Limit define which sub-sequence of matching entities
  323. to return. Start and End take cursors, Offset and Limit take integers. Start
  324. and Offset affect the first result, End and Limit affect the last result.
  325. If both Start and Offset are set, then the offset is relative to Start.
  326. If both End and Limit are set, then the earliest constraint wins. Limit is
  327. relative to Start+Offset, not relative to End. As a special case, a
  328. negative limit means unlimited.
  329. Example code:
  330. type Widget struct {
  331. Description string
  332. Price int
  333. }
  334. func printWidgets(ctx context.Context, client *datastore.Client) {
  335. q := datastore.NewQuery("Widget").
  336. Filter("Price <", 1000).
  337. Order("-Price")
  338. t := client.Run(ctx, q)
  339. for {
  340. var x Widget
  341. key, err := t.Next(&x)
  342. if err == iterator.Done {
  343. break
  344. }
  345. if err != nil {
  346. // Handle error.
  347. }
  348. fmt.Printf("Key=%v\nWidget=%#v\n\n", key, x)
  349. }
  350. }
  351. Transactions
  352. Client.RunInTransaction runs a function in a transaction.
  353. Example code:
  354. type Counter struct {
  355. Count int
  356. }
  357. func incCount(ctx context.Context, client *datastore.Client) {
  358. var count int
  359. key := datastore.NameKey("Counter", "singleton", nil)
  360. _, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
  361. var x Counter
  362. if err := tx.Get(key, &x); err != nil && err != datastore.ErrNoSuchEntity {
  363. return err
  364. }
  365. x.Count++
  366. if _, err := tx.Put(key, &x); err != nil {
  367. return err
  368. }
  369. count = x.Count
  370. return nil
  371. })
  372. if err != nil {
  373. // Handle error.
  374. }
  375. // The value of count is only valid once the transaction is successful
  376. // (RunInTransaction has returned nil).
  377. fmt.Printf("Count=%d\n", count)
  378. }
  379. Pass the ReadOnly option to RunInTransaction if your transaction is used only for Get,
  380. GetMulti or queries. Read-only transactions are more efficient.
  381. Google Cloud Datastore Emulator
  382. This package supports the Cloud Datastore emulator, which is useful for testing and
  383. development. Environment variables are used to indicate that datastore traffic should be
  384. directed to the emulator instead of the production Datastore service.
  385. To install and set up the emulator and its environment variables, see the documentation
  386. at https://cloud.google.com/datastore/docs/tools/datastore-emulator.
  387. */
  388. package datastore // import "cloud.google.com/go/datastore"