Proposed resolution:

This wording is relative to N5032.

[Drafting Note: Two mutually exclusive options are prepared, depicted below by Option A and Option B, respectively.]

Option A: Forbid program-defined std::optional<T> specializations

1. Modify [optional.optional.general] as indicated:

   -2- A type `X` is a valid contained type for `optional` […]. If `T` is an object type, `T` shall meet the Cpp17Destructible requirements (Table 35).

   -?- The behavior of a program that adds a specialization for `optional` is undefined.

Option B: Add a std::optional<T> constructor taking an invocable

1. Modify [utility.syn], header <utility> synopsis, as indicated:

   […]
   namespace std {
     […]
     template<size_t I>
       struct in_place_index_t {
         explicit in_place_index_t() = default;
       };
     template<size_t I> constexpr in_place_index_t<I> in_place_index{};
     
     // construction from arbitrary initializers
     
     struct from_continuation_t {
       explicit from_continuation_t() = default;
     };
     inline constexpr from_continuation_t from_continuation{};
     […]
   }
   

2. Modify [optional.optional.general] as indicated:

   namespace std {
     template<class T>
     class optional {
     public:
       […]
       // [optional.ctor], constructors
       constexpr optional() noexcept;
       constexpr optional(nullopt_t) noexcept;
       […]
       template<class... Args>
         constexpr explicit optional(in_place_t, Args&&...);
       template<class U, class... Args>
         constexpr explicit optional(in_place_t, initializer_list<U>, Args&&...);
       template<class F, class... Args>
         constexpr explicit optional(from_continuation_t, F&&, Args&&...);
       template<class U = remove_cv_t<T>>
         constexpr explicit(see below) optional(U&&);
       […]
     };
     […]
   }
   

3. Modify [optional.ctor] as indicated:

   template<class U, class... Args>
     constexpr explicit optional(in_place_t, initializer_list<U> il, Args&&... args);
   

   -18- Constraints: […]

   […]

   -22- Remarks: If `T`'s constructor selected for the initialization is a constexpr constructor, this constructor is a constexpr constructor.

   template<class F, class... Args>
     constexpr explicit optional(from_continuation_t, F&& f, Args&&... args);
   

   -?- Mandates: decltype(std::invoke(std::forward<F>(f), std::forward<Args>(args)...)) is `T`.

   -?- Effects: Direct-non-list-initializes val with std::invoke(std::forward<F>(f), std::forward<Args>(args)...).

   -?- Postconditions: `*this` contains a value.

4. Modify [optional.optional.ref.general] as indicated:

   namespace std {
     template<class T>
     class optional<T&> {
     public:
       […]
       // [optional.ref.ctor], constructors
       constexpr optional() noexcept = default;
       constexpr optional(nullopt_t) noexcept : optional() {}
       […]
       template<class Arg>
         constexpr explicit optional(in_place_t, Arg&& arg);
       template<class F, class... Args>
         constexpr explicit optional(from_continuation_t, F&& f, Args&&... args);
       template<class U>
         constexpr explicit(see below) optional(U&& u) noexcept(see below);
       […]
     };
     […]
   }
   

5. Modify [optional.ref.ctor] as indicated:

   template<class U, class Arg>
     constexpr explicit optional(in_place_t, Arg&& arg);
   

   -1- Constraints: […]

   -2- Effects: […]

   -3- Postconditions: […]

   template<class F, class Arg>
     constexpr explicit optional(from_continuation_t, F&& f, Arg&& arg);
   

   -?- Mandates: decltype(std::invoke(std::forward<F>(f), std::forward<Args>(args)...)) is T&.

   -?- Effects: Equivalent to: convert-ref-init-val(std::invoke(std::forward<F>(f), std::forward<Args>(args)...)).

   -?- Postconditions: `*this` contains a value.

Currently (that is, as of the draft at N5032), [optional.monadic] specifies that std::optional<T>::transform(F&&f)& shall do the following (and similar for the other overloads):

Let `U` be remove_cv_t<invoke_result_t<F, decltype((val))>>.

Mandates: […]

[Note 1: There is no requirement that `U` is movable ([dcl.init.general]). — end note]

Returns: If `*this` contains a value, an optional<U> object whose contained value is direct-non-list-initialized with invoke(std::forward<F>(f), val); otherwise, optional<U>().

However, none of the standard constructors or other member functions of optional<U> provide a surefire way to initialize the contained `U` value with an expression like invoke(std::forward<F>(f), val). The closest are the `in_place_t`/`emplace` overloads, which almost but not quite admit a generic implementation of `transform`. This looks roughly like:

namespace std {
  template<class _F> struct __later {
    _F __f;
    operator decltype(std::move(__f)())() && { return std::move(__f)(); }
  };

  template<class _T> class optional {
    // etc.
  public:
    template<class _F> constexpr auto transform(_F &&__f) & {
      using _U = remove_cv_t<invoke_result_t<_F, _T&>>;
      if(!has_value()) return optional<_U>();
      return optional<_U>(in_place, __later([&] -> _U {
        return std::invoke(std::forward<_F>(__f), value());
      }));
    }
  };
}

Unfortunately, this does not quite meet the specification. The issue is if `U` is a type with a U(auto&&) constructor:

struct oops {
  oops() = default;
  oops(auto&&) { std::cout << "launching missiles\n"; }
};

int main() {
  std::optional<int> oi(5);
  oi.transform([](auto& i) { return oops(); });
  // missiles get launched when they shouldn't
}

In this case, the rules for direct-initialization (see [dcl.init] bullet 16.6.2) will select the template constructor over the conversion function on the `__later` specialization. [Complete example 1]

To avoid this problem, standard library implementors generally implement std::optional<T>::transform with a non-standard constructor on their std::optional<T> primary template; roughly:

namespace std {
  struct __optional_from_invocable_tag {
    constexpr explicit __optional_from_invocable_tag() { }
  };
  
  template<typename _T>
  class optional {
    // etc.
  public:
    template<typename _F, typename _V>
    constexpr optional(__optional_from_invocable_tag, _F &&__f, _V &&__v)
      : __present(true)
      , __val(std::invoke(std::forward<_F>(__f), std::forward<_V>(__v)))
    { }

    template<class _F> constexpr auto transform(_F &&__f) & {
      using _U = remove_cv_t<invoke_result_t<_F, _T&>>;
      if(!has_value()) return optional<_U>();
      return optional<_U>(
        __optional_from_invocable_tag(),
        std::forward<_F>(__f), value());
    }
  };
}

[Complete example 2]. Note that the missiles are not launched.

Now for the real issue: if a user program wants to specialize `std::optional` for a program-defined type, it will have to explicitly rely on these details of its standard library implementation in order to be supported by the standard library's `transform` implementation. Specifically, it will have to provide a non-standard constructor with a signature matching the library implementation's expectations. (A portable implementation of `transform` itself is more-or-less possible for a program-defined specialization by using a circumlocution like std::optional<std::monostate>(std::in_place).transform(/* ... */).)

The root problem is that the standard interface of std::optional<U> provides for direct-initialization of the contained `U` by arbitrary glvalues, but not by an arbitrary prvalue (that is, by calling an arbitrary invocable). This forces library implementations to invent their own non-standard interfaces for doing so, which then makes it impossible for those implementations to support program-defined specializations of `std::optional` that only meet the minimal requirements of the standard, and do not support those non-standard interfaces.

The fact that std::optional<T>::transform makes implementing `std::optional` while supporting program-defined specializations basically impossible does not appear to be intentional. P0798R8, which introduced std::optional<T>::transform, does not mention this side-effect of its standardization.

There are at least two different resolutions that immediately come to mind.

Option A: Forbid program-defined std::optional<T> specializations

Taking this option would immediately solve the problem. However, in my opinion, this would be unnecessarily restrictive. Specializing `std::optional` is a useful thing to allow, as it allows replacing the common struct optional<T> { union { T val; }; bool present; } representation with something more compact when `T` has unused values/unused bits.

Option B: Add a std::optional<T> constructor taking an invocable

This option more-or-less formalizes existing practice, using a type tag to gate the new constructor. It would be ideal to extend this idea to `emplace` and then to the various `in_place_t` constructors and `emplace` functions in other parts of the standard, but the wording presented here is restricted to fixing this issue.

Changing std::optional<T&> doesn't seem strictly necessary, but introducing a nonuniformity seems like a bad idea. I'm not 100% certain about the wording for the new constructors.